From a3a1c5d1867977c87330e287517111a983f70ef8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 2 Nov 2017 22:12:17 +0000 Subject: [PATCH 001/511] Initial commit: add license. --- LICENSE | 661 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 661 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..be3f7b2 --- /dev/null +++ b/LICENSE @@ -0,0 +1,661 @@ + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. From 932d3803ed34838581b6efd14f6a6f2cc713e4df Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 2 Nov 2017 22:20:55 +0000 Subject: [PATCH 002/511] Initial commit of the actual server code. --- .gitignore | 9 +++ etesync_server/__init__.py | 0 etesync_server/settings.py | 129 +++++++++++++++++++++++++++++++++++++ etesync_server/urls.py | 40 ++++++++++++ etesync_server/wsgi.py | 16 +++++ manage.py | 22 +++++++ requirements.txt | 5 ++ 7 files changed, 221 insertions(+) create mode 100644 .gitignore create mode 100644 etesync_server/__init__.py create mode 100644 etesync_server/settings.py create mode 100644 etesync_server/urls.py create mode 100644 etesync_server/wsgi.py create mode 100755 manage.py create mode 100644 requirements.txt diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..10f1650 --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +/db.sqlite3 +Session.vim +/local_settings.py +/.venv +/.coverage +/htmlcov + +__pycache__ +.*.swp diff --git a/etesync_server/__init__.py b/etesync_server/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/etesync_server/settings.py b/etesync_server/settings.py new file mode 100644 index 0000000..fd1ae53 --- /dev/null +++ b/etesync_server/settings.py @@ -0,0 +1,129 @@ +""" +Django settings for etesync_server project. + +Generated by 'django-admin startproject' using Django 1.10.6. + +For more information on this file, see +https://docs.djangoproject.com/en/1.10/topics/settings/ + +For the full list of settings and their values, see +https://docs.djangoproject.com/en/1.10/ref/settings/ +""" + +import os + +# Build paths inside the project like this: os.path.join(BASE_DIR, ...) +BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + +# Quick-start development settings - unsuitable for production +# See https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/ + +# SECURITY WARNING: keep the secret key used in production secret! +SECRET_KEY = '' + +# SECURITY WARNING: don't run with debug turned on in production! +DEBUG = False + +ALLOWED_HOSTS = [] + + +# Application definition + +INSTALLED_APPS = [ + 'django.contrib.admin', + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', + 'rest_framework', + 'rest_framework.authtoken', + 'journal.apps.JournalConfig', +] + +MIDDLEWARE = [ + 'django.middleware.security.SecurityMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.common.CommonMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', + 'django.middleware.clickjacking.XFrameOptionsMiddleware', +] + +ROOT_URLCONF = 'etesync_server.urls' + +TEMPLATES = [ + { + 'BACKEND': 'django.template.backends.django.DjangoTemplates', + 'DIRS': [], + 'APP_DIRS': True, + 'OPTIONS': { + 'context_processors': [ + 'django.template.context_processors.debug', + 'django.template.context_processors.request', + 'django.contrib.auth.context_processors.auth', + 'django.contrib.messages.context_processors.messages', + ], + }, + }, +] + +WSGI_APPLICATION = 'etesync_server.wsgi.application' + + +# Database +# https://docs.djangoproject.com/en/1.10/ref/settings/#databases + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.environ.get('ETESYNC_DB_PATH', + os.path.join(BASE_DIR, 'db.sqlite3')), + } +} + + +# Password validation +# https://docs.djangoproject.com/en/1.10/ref/settings/#auth-password-validators + +AUTH_PASSWORD_VALIDATORS = [ + { + 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', # noqa + }, + { + 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', # noqa + }, + { + 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', # noqa + }, + { + 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', # noqa + }, +] + + +# Internationalization +# https://docs.djangoproject.com/en/1.10/topics/i18n/ + +LANGUAGE_CODE = 'en-us' + +TIME_ZONE = 'UTC' + +USE_I18N = True + +USE_L10N = True + +USE_TZ = True + + +# Static files (CSS, JavaScript, Images) +# https://docs.djangoproject.com/en/1.10/howto/static-files/ + +STATIC_URL = '/static/' + + +JOURNAL_API_PERMISSIONS = ( + 'rest_framework.permissions.IsAuthenticated', + ) diff --git a/etesync_server/urls.py b/etesync_server/urls.py new file mode 100644 index 0000000..ebde04a --- /dev/null +++ b/etesync_server/urls.py @@ -0,0 +1,40 @@ +"""etesync_server URL Configuration + +The `urlpatterns` list routes URLs to views. For more information please see: + https://docs.djangoproject.com/en/1.10/topics/http/urls/ +Examples: +Function views + 1. Add an import: from my_app import views + 2. Add a URL to urlpatterns: url(r'^$', views.home, name='home') +Class-based views + 1. Add an import: from other_app.views import Home + 2. Add a URL to urlpatterns: url(r'^$', Home.as_view(), name='home') +Including another URLconf + 1. Import the include() function: from django.conf.urls import url, include + 2. Add a URL to urlpatterns: url(r'^blog/', include('blog.urls')) +""" +from django.conf.urls import include, url +from django.contrib import admin + +from rest_framework_nested import routers +from rest_framework.authtoken import views as token_views + +from journal import views + +router = routers.DefaultRouter() +router.register(r'journals', views.JournalViewSet) +router.register(r'journal/(?P[^/]+)', views.EntryViewSet) +router.register(r'user', views.UserInfoViewSet) + +journals_router = routers.NestedSimpleRouter(router, r'journals', lookup='journal') +journals_router.register(r'members', views.MembersViewSet, base_name='journal-members') +journals_router.register(r'entries', views.EntryViewSet, base_name='journal-entries') + + +urlpatterns = [ + url(r'^api/v1/', include(router.urls)), + url(r'^api/v1/', include(journals_router.urls)), + url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), + url(r'^api-token-auth/', token_views.obtain_auth_token), + url(r'^admin/', admin.site.urls), +] diff --git a/etesync_server/wsgi.py b/etesync_server/wsgi.py new file mode 100644 index 0000000..eeb40bc --- /dev/null +++ b/etesync_server/wsgi.py @@ -0,0 +1,16 @@ +""" +WSGI config for etesync_server project. + +It exposes the WSGI callable as a module-level variable named ``application``. + +For more information on this file, see +https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/ +""" + +import os + +from django.core.wsgi import get_wsgi_application + +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etesync_server.settings") + +application = get_wsgi_application() diff --git a/manage.py b/manage.py new file mode 100755 index 0000000..56f041a --- /dev/null +++ b/manage.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python +import os +import sys + +if __name__ == "__main__": + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etesync_server.settings") + try: + from django.core.management import execute_from_command_line + except ImportError: + # The above import may fail for some other reason. Ensure that the + # issue is really that Django is missing to avoid masking other + # exceptions on Python 2. + try: + import django + except ImportError: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) + raise + execute_from_command_line(sys.argv) diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..71ee320 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,5 @@ +Django==1.11.7 +djangorestframework==3.7.1 +drf-nested-routers==0.90.0 +pytz==2017.3 +git+git://github.com/etesync/journal-manager@v0.4.1 From 64116e58a57839283f1bf197ffbf4f7fb093c1f9 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 19 Nov 2017 18:01:34 +0000 Subject: [PATCH 003/511] Add a readme. --- README.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..86b3870 --- /dev/null +++ b/README.md @@ -0,0 +1,51 @@ +A skeleton app for running your own [EteSync](https://www.etesync.com) server + +# Installation + +To setup your own EteSync server based on the git version just clone this +git repository and set up the django app: + +``` +git clone https://github.com/etesync/server-skeleton.git + +cd server-skeleton + +# Set up the environment and deps +virtualenv .venv +source .venv/bin/activate + +pip install -r requirements.txt +``` + +Set the django ```SECRET_KEY``` and ```ALLOWED_HOSTS``` in [the settings file](etesync_server/settings.py). +For more information on these please refer to the [django deployment checklist](https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/). + +Now you can initialise our django app + +``` +./manage.py migrate +``` + +And you are done! You can now either run the debug server just to see everything works as expected by running: + +``` +./manage.py runserver 0.0.0.0:8000 +``` + +Using the debug server is production is not recommended, so you should configure your webserver to serve +etesync (with TLS). An example on how to do so with nginx can be found [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). + +# Usage + +Create yourself an admin user: + +``` +./manage.py createsuperuser +``` + +At this stage you can either just use the admin user, or better yet, go to: ```www.your-etesync-install.com/admin``` +and create a non-privileged user that you can use. + +That's it! + +Now all that's left is to open the EteSync app, add an account, and set your custom server address under the "advance" section. From e57279149dc183966da85e64560f1b939089a308 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 19 Nov 2017 21:59:25 +0000 Subject: [PATCH 004/511] README: fix typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 86b3870..63f5926 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ And you are done! You can now either run the debug server just to see everything ./manage.py runserver 0.0.0.0:8000 ``` -Using the debug server is production is not recommended, so you should configure your webserver to serve +Using the debug server in production is not recommended, so you should configure your webserver to serve etesync (with TLS). An example on how to do so with nginx can be found [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). # Usage From cea4f318cabcd1637509cdb966221b6eaf216d49 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 29 Nov 2017 17:59:59 +0000 Subject: [PATCH 005/511] README: Add a note about supporting etesync. --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 63f5926..51a3c07 100644 --- a/README.md +++ b/README.md @@ -49,3 +49,7 @@ and create a non-privileged user that you can use. That's it! Now all that's left is to open the EteSync app, add an account, and set your custom server address under the "advance" section. + +# Supporting EteSync + +Please consider registering an account even if you self-host in order to support the development of EteSync, or help by spreading the word. From edbd28b67a14dd7f007bbf45f2856859ab2026dc Mon Sep 17 00:00:00 2001 From: x11x <28614156+x11x@users.noreply.github.com> Date: Sun, 18 Feb 2018 13:35:23 +1000 Subject: [PATCH 006/511] Change requirements.txt to allow updating to latest patch version Also, allow any version of pytz --- requirements.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/requirements.txt b/requirements.txt index 71ee320..78f52df 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ -Django==1.11.7 -djangorestframework==3.7.1 +Django>=1.11,<1.11.999 +djangorestframework>=3.7,<3.7.999 drf-nested-routers==0.90.0 -pytz==2017.3 +pytz git+git://github.com/etesync/journal-manager@v0.4.1 From 276a926fcbd2a03fb8775fad27ca8e2b9d5261eb Mon Sep 17 00:00:00 2001 From: x11x <28614156+x11x@users.noreply.github.com> Date: Sun, 18 Feb 2018 13:43:29 +1000 Subject: [PATCH 007/511] Use secret.txt file auto-generated in project root as default SECRET_KEY Also add it to .gitignore --- .gitignore | 2 ++ etesync_server/settings.py | 11 ++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 10f1650..aa7817e 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,8 @@ Session.vim /.venv /.coverage /htmlcov +/secret.txt +/static __pycache__ .*.swp diff --git a/etesync_server/settings.py b/etesync_server/settings.py index fd1ae53..ad433cb 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -11,6 +11,7 @@ https://docs.djangoproject.com/en/1.10/ref/settings/ """ import os +from django.core.management import utils # Build paths inside the project like this: os.path.join(BASE_DIR, ...) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -20,7 +21,15 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # See https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/ # SECURITY WARNING: keep the secret key used in production secret! -SECRET_KEY = '' +SECRET_KEY_FILE = os.path.join(BASE_DIR, "secret.txt") + +try: + with open(SECRET_KEY_FILE, "r") as f: + SECRET_KEY = f.read().strip() +except EnvironmentError: + with open(SECRET_KEY_FILE, "w") as f: + SECRET_KEY = utils.get_random_secret_key() + f.write(SECRET_KEY) # SECURITY WARNING: don't run with debug turned on in production! DEBUG = False From 69655008e07ec0818c5293289fd08676d6d6e131 Mon Sep 17 00:00:00 2001 From: x11x <28614156+x11x@users.noreply.github.com> Date: Sun, 18 Feb 2018 14:19:29 +1000 Subject: [PATCH 008/511] Refactor out the secret.txt file handling to a utils module --- etesync_server/settings.py | 14 ++++---------- etesync_server/utils.py | 11 +++++++++++ 2 files changed, 15 insertions(+), 10 deletions(-) create mode 100644 etesync_server/utils.py diff --git a/etesync_server/settings.py b/etesync_server/settings.py index ad433cb..fe65225 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -11,7 +11,7 @@ https://docs.djangoproject.com/en/1.10/ref/settings/ """ import os -from django.core.management import utils +from .utils import get_secret_from_file # Build paths inside the project like this: os.path.join(BASE_DIR, ...) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -21,15 +21,9 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # See https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/ # SECURITY WARNING: keep the secret key used in production secret! -SECRET_KEY_FILE = os.path.join(BASE_DIR, "secret.txt") - -try: - with open(SECRET_KEY_FILE, "r") as f: - SECRET_KEY = f.read().strip() -except EnvironmentError: - with open(SECRET_KEY_FILE, "w") as f: - SECRET_KEY = utils.get_random_secret_key() - f.write(SECRET_KEY) +# See secret.py for how this is generated; uses a file 'secret.txt' in the root +# directory +SECRET_KEY = get_secret_from_file(os.path.join(BASE_DIR, "secret.txt")) # SECURITY WARNING: don't run with debug turned on in production! DEBUG = False diff --git a/etesync_server/utils.py b/etesync_server/utils.py new file mode 100644 index 0000000..8f85f10 --- /dev/null +++ b/etesync_server/utils.py @@ -0,0 +1,11 @@ +from django.core.management import utils + +def get_secret_from_file(path): + try: + with open(path, "r") as f: + return f.read().strip() + except EnvironmentError: + with open(path, "w") as f: + secret_key = utils.get_random_secret_key() + f.write(secret_key) + return secret_key From 3ee704bfb101cbb40b962da5e55b1cb13d0054b1 Mon Sep 17 00:00:00 2001 From: x11x <28614156+x11x@users.noreply.github.com> Date: Sun, 18 Feb 2018 14:20:29 +1000 Subject: [PATCH 009/511] README: elaborate on settings, provide docs links, explain 'secret.txt' Also describe how to update to latest patch-level versions. --- README.md | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 51a3c07..571fc4d 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,20 @@ source .venv/bin/activate pip install -r requirements.txt ``` -Set the django ```SECRET_KEY``` and ```ALLOWED_HOSTS``` in [the settings file](etesync_server/settings.py). -For more information on these please refer to the [django deployment checklist](https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/). +Edit the [settings file](etesync_server/settings.py). Please refer to the +[Django deployment +checklist](https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/) +for full instructions on how to configure a Django app for production. Some +particular settings that should be edited are: + * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) + -- this is the list of host/domain names or addresses on which the app +will be served + * [`DEBUG`](https://docs.djangoproject.com/en/1.11/ref/settings/#debug) + -- handy for debugging, set to `False` for production + * [`SECRET_KEY`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-SECRET_KEY) + -- an ephemeral secret used for various cryptographic signing and token +generation purposes. See below for how default configuration of +`SECRET_KEY` works for this project. Now you can initialise our django app @@ -50,6 +62,26 @@ That's it! Now all that's left is to open the EteSync app, add an account, and set your custom server address under the "advance" section. +# `SECRET_KEY` and `secret.txt` + +The default configuration creates a file “`secret.txt`” in the project’s +base directory, which is used as the value of the Django `SECRET_KEY` +setting. You can revoke this key by deleting the `secret.txt` file and the +next time the app is run, a new one will be generated. Make sure you keep +the `secret.txt` file secret (don’t accidentally commit it to version +control, exclude it from your backups, etc.). If you want to change to a +more secure system for storing secrets, edit `etesync_server/settings.py` +and implement your own method for setting `SECRET_KEY` (remove the line +where it uses the `get_secret_from_file` function). Read the Django docs +for more information about the `SECRET_KEY` and its uses. + +# Updating + +Inside the virtualenv, run `pip install -U -r requirements.txt` to update +dependencies to latest compatible versions of Django and +djangorestframework (it will only update to latest patch level which should +be API-compatible). + # Supporting EteSync Please consider registering an account even if you self-host in order to support the development of EteSync, or help by spreading the word. From c93fde3ddc39c55abc2151e11d8d2c949f5ff406 Mon Sep 17 00:00:00 2001 From: x11x <28614156+x11x@users.noreply.github.com> Date: Sun, 18 Feb 2018 14:30:50 +1000 Subject: [PATCH 010/511] README: update Django docs link version to 1.11 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 571fc4d..2f80211 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ pip install -r requirements.txt Edit the [settings file](etesync_server/settings.py). Please refer to the [Django deployment -checklist](https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/) +checklist](https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/) for full instructions on how to configure a Django app for production. Some particular settings that should be edited are: * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) From 3e3d4c04544d329d615581688116781c5195ae31 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 20 Mar 2018 17:45:33 +0000 Subject: [PATCH 011/511] README: Add info about librepay. --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 2f80211..525d220 100644 --- a/README.md +++ b/README.md @@ -85,3 +85,7 @@ be API-compatible). # Supporting EteSync Please consider registering an account even if you self-host in order to support the development of EteSync, or help by spreading the word. + +Additionally, following user requests, we've opened an account on Librepay and accept donations there: + +[![Donate using Librepay](https://liberapay.com/assets/widgets/donate.svg)](https://liberapay.com/EteSync/donate) From 27d56ee87be942ec435df44857222ba1761c7d82 Mon Sep 17 00:00:00 2001 From: Victor Rezende dos Santos Date: Tue, 1 May 2018 12:46:50 -0300 Subject: [PATCH 012/511] Update Dependencies * Updated etesync/journal-manager to v0.5.2 * Updated Django to 2.0.5 - Updated documentation links - Changed syntax and imports of `urls.py` * Updated Django REST Framework to 3.8.2 * Updated DFR Nested Routers to 0.90.2 --- etesync_server/settings.py | 24 ++++++++++++------------ etesync_server/urls.py | 22 +++++++++++----------- etesync_server/wsgi.py | 2 +- requirements.txt | 8 ++++---- 4 files changed, 28 insertions(+), 28 deletions(-) diff --git a/etesync_server/settings.py b/etesync_server/settings.py index fe65225..bc44337 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -1,13 +1,13 @@ """ Django settings for etesync_server project. -Generated by 'django-admin startproject' using Django 1.10.6. +Generated by 'django-admin startproject' using Django 2.0.5. For more information on this file, see -https://docs.djangoproject.com/en/1.10/topics/settings/ +https://docs.djangoproject.com/en/2.0/topics/settings/ For the full list of settings and their values, see -https://docs.djangoproject.com/en/1.10/ref/settings/ +https://docs.djangoproject.com/en/2.0/ref/settings/ """ import os @@ -18,7 +18,7 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # Quick-start development settings - unsuitable for production -# See https://docs.djangoproject.com/en/1.10/howto/deployment/checklist/ +# See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/ # SECURITY WARNING: keep the secret key used in production secret! # See secret.py for how this is generated; uses a file 'secret.txt' in the root @@ -77,7 +77,7 @@ WSGI_APPLICATION = 'etesync_server.wsgi.application' # Database -# https://docs.djangoproject.com/en/1.10/ref/settings/#databases +# https://docs.djangoproject.com/en/2.0/ref/settings/#databases DATABASES = { 'default': { @@ -89,26 +89,26 @@ DATABASES = { # Password validation -# https://docs.djangoproject.com/en/1.10/ref/settings/#auth-password-validators +# https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators AUTH_PASSWORD_VALIDATORS = [ { - 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', # noqa + 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', }, { - 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', # noqa + 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', }, { - 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', # noqa + 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', }, { - 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', # noqa + 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', }, ] # Internationalization -# https://docs.djangoproject.com/en/1.10/topics/i18n/ +# https://docs.djangoproject.com/en/2.0/topics/i18n/ LANGUAGE_CODE = 'en-us' @@ -122,7 +122,7 @@ USE_TZ = True # Static files (CSS, JavaScript, Images) -# https://docs.djangoproject.com/en/1.10/howto/static-files/ +# https://docs.djangoproject.com/en/2.0/howto/static-files/ STATIC_URL = '/static/' diff --git a/etesync_server/urls.py b/etesync_server/urls.py index ebde04a..c76d88e 100644 --- a/etesync_server/urls.py +++ b/etesync_server/urls.py @@ -1,19 +1,19 @@ """etesync_server URL Configuration The `urlpatterns` list routes URLs to views. For more information please see: - https://docs.djangoproject.com/en/1.10/topics/http/urls/ + https://docs.djangoproject.com/en/2.0/topics/http/urls/ Examples: Function views 1. Add an import: from my_app import views - 2. Add a URL to urlpatterns: url(r'^$', views.home, name='home') + 2. Add a URL to urlpatterns: path('', views.home, name='home') Class-based views 1. Add an import: from other_app.views import Home - 2. Add a URL to urlpatterns: url(r'^$', Home.as_view(), name='home') + 2. Add a URL to urlpatterns: path('', Home.as_view(), name='home') Including another URLconf - 1. Import the include() function: from django.conf.urls import url, include - 2. Add a URL to urlpatterns: url(r'^blog/', include('blog.urls')) + 1. Import the include() function: from django.urls import include, path + 2. Add a URL to urlpatterns: path('blog/', include('blog.urls')) """ -from django.conf.urls import include, url +from django.urls import include, path, re_path from django.contrib import admin from rest_framework_nested import routers @@ -32,9 +32,9 @@ journals_router.register(r'entries', views.EntryViewSet, base_name='journal-entr urlpatterns = [ - url(r'^api/v1/', include(router.urls)), - url(r'^api/v1/', include(journals_router.urls)), - url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), - url(r'^api-token-auth/', token_views.obtain_auth_token), - url(r'^admin/', admin.site.urls), + re_path(r'^api/v1/', include(router.urls)), + re_path(r'^api/v1/', include(journals_router.urls)), + re_path(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), + re_path(r'^api-token-auth/', token_views.obtain_auth_token), + path('admin/', admin.site.urls), ] diff --git a/etesync_server/wsgi.py b/etesync_server/wsgi.py index eeb40bc..6738999 100644 --- a/etesync_server/wsgi.py +++ b/etesync_server/wsgi.py @@ -4,7 +4,7 @@ WSGI config for etesync_server project. It exposes the WSGI callable as a module-level variable named ``application``. For more information on this file, see -https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/ +https://docs.djangoproject.com/en/2.0/howto/deployment/wsgi/ """ import os diff --git a/requirements.txt b/requirements.txt index 78f52df..03d8859 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ -Django>=1.11,<1.11.999 -djangorestframework>=3.7,<3.7.999 -drf-nested-routers==0.90.0 +Django>=2.0.5,<2.0.999 +djangorestframework>=3.8.2,<3.8.999 +drf-nested-routers==0.90.2 pytz -git+git://github.com/etesync/journal-manager@v0.4.1 +git+git://github.com/etesync/journal-manager@v0.5.2 From dfc7c8b1639710a6a03712c65c7ef14dd2a4237b Mon Sep 17 00:00:00 2001 From: Nuntius Date: Thu, 28 Jun 2018 15:05:45 +0200 Subject: [PATCH 013/511] Mention setting up CORS in Readme --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 525d220..8b0a26e 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,8 @@ And you are done! You can now either run the debug server just to see everything Using the debug server in production is not recommended, so you should configure your webserver to serve etesync (with TLS). An example on how to do so with nginx can be found [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). +If you plan to use the webclient, you also want to enable CORS in your nginx config. You can find a "wide +open CORS config" example [here](https://enable-cors.org/server_nginx.html) to be adjusted to your needs. # Usage From de4f49973f08027f0818c5780dd65517eb3c575e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 28 Jun 2018 16:37:21 +0100 Subject: [PATCH 014/511] Disable CORS altogether. This solves #4 better than dfc7c8b1639710a6a03712c65c7ef14dd2a4237b --- etesync_server/settings.py | 5 +++++ requirements.txt | 1 + 2 files changed, 6 insertions(+) diff --git a/etesync_server/settings.py b/etesync_server/settings.py index bc44337..521cb2f 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -40,6 +40,7 @@ INSTALLED_APPS = [ 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', + 'corsheaders', 'rest_framework', 'rest_framework.authtoken', 'journal.apps.JournalConfig', @@ -48,6 +49,7 @@ INSTALLED_APPS = [ MIDDLEWARE = [ 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', + 'corsheaders.middleware.CorsMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', @@ -130,3 +132,6 @@ STATIC_URL = '/static/' JOURNAL_API_PERMISSIONS = ( 'rest_framework.permissions.IsAuthenticated', ) + +# Cors +CORS_ORIGIN_ALLOW_ALL = True diff --git a/requirements.txt b/requirements.txt index 03d8859..65d8a96 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,5 @@ Django>=2.0.5,<2.0.999 +django-cors-headers==2.3.0 djangorestframework>=3.8.2,<3.8.999 drf-nested-routers==0.90.2 pytz From 2756f30eb4d984a6913a20fefb76729792abb40f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 28 Jun 2018 16:39:48 +0100 Subject: [PATCH 015/511] Revert "Mention setting up CORS in Readme" This is better solved by de4f49973f08027f0818c5780dd65517eb3c575e This reverts commit dfc7c8b1639710a6a03712c65c7ef14dd2a4237b. --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 8b0a26e..525d220 100644 --- a/README.md +++ b/README.md @@ -46,8 +46,6 @@ And you are done! You can now either run the debug server just to see everything Using the debug server in production is not recommended, so you should configure your webserver to serve etesync (with TLS). An example on how to do so with nginx can be found [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). -If you plan to use the webclient, you also want to enable CORS in your nginx config. You can find a "wide -open CORS config" example [here](https://enable-cors.org/server_nginx.html) to be adjusted to your needs. # Usage From 19c49975d7ade4d19991025c656e737a1922361f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 29 Aug 2018 19:18:38 +0100 Subject: [PATCH 016/511] README: remove the now defunct librepay --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 525d220..2f80211 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,3 @@ be API-compatible). # Supporting EteSync Please consider registering an account even if you self-host in order to support the development of EteSync, or help by spreading the word. - -Additionally, following user requests, we've opened an account on Librepay and accept donations there: - -[![Donate using Librepay](https://liberapay.com/assets/widgets/donate.svg)](https://liberapay.com/EteSync/donate) From 0738d0246689a477b2b496f034701021249a0cec Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 16 Sep 2018 20:15:49 +0100 Subject: [PATCH 017/511] README: make the usage of virtualenv3 more explicit --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2f80211..d390022 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ git clone https://github.com/etesync/server-skeleton.git cd server-skeleton # Set up the environment and deps -virtualenv .venv +virtualenv3 .venv source .venv/bin/activate pip install -r requirements.txt From f1bfaf024267d7835422babe39aa1b1160440635 Mon Sep 17 00:00:00 2001 From: Ben Boeckel Date: Mon, 29 Oct 2018 18:23:43 -0400 Subject: [PATCH 018/511] settings: split SECRET_FILE from SECRET_KEY --- etesync_server/settings.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/etesync_server/settings.py b/etesync_server/settings.py index 521cb2f..9d5f119 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -23,7 +23,8 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # SECURITY WARNING: keep the secret key used in production secret! # See secret.py for how this is generated; uses a file 'secret.txt' in the root # directory -SECRET_KEY = get_secret_from_file(os.path.join(BASE_DIR, "secret.txt")) +SECRET_FILE = os.path.join(BASE_DIR, "secret.txt") +SECRET_KEY = get_secret_from_file(SECRET_FILE) # SECURITY WARNING: don't run with debug turned on in production! DEBUG = False From 872af56e2cd00669f13a80c74710bf7b9a19940e Mon Sep 17 00:00:00 2001 From: Ben Boeckel Date: Mon, 29 Oct 2018 18:25:35 -0400 Subject: [PATCH 019/511] settings: support an external settings file Fixes #8 --- etesync_server/settings.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/etesync_server/settings.py b/etesync_server/settings.py index 9d5f119..d4b93d8 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -136,3 +136,9 @@ JOURNAL_API_PERMISSIONS = ( # Cors CORS_ORIGIN_ALLOW_ALL = True + +# Make an `etesync_site_settings` module available to override settings. +try: + from etesync_site_settings import * +except ImportError: + pass From e02fa53c34200de59288a16ed00fa0621a007578 Mon Sep 17 00:00:00 2001 From: Ben Boeckel Date: Mon, 29 Oct 2018 18:26:13 -0400 Subject: [PATCH 020/511] settings: defer SECRET_KEY until after site settings This allows overriding just `SECRET_FILE` for a custom secret path. --- etesync_server/settings.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/etesync_server/settings.py b/etesync_server/settings.py index d4b93d8..9556167 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -24,7 +24,6 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # See secret.py for how this is generated; uses a file 'secret.txt' in the root # directory SECRET_FILE = os.path.join(BASE_DIR, "secret.txt") -SECRET_KEY = get_secret_from_file(SECRET_FILE) # SECURITY WARNING: don't run with debug turned on in production! DEBUG = False @@ -142,3 +141,6 @@ try: from etesync_site_settings import * except ImportError: pass + +if 'SECRET_KEY' not in locals(): + SECRET_KEY = get_secret_from_file(SECRET_FILE) From 8f50a69b39bef7f421590e3de26b822df0ceaf6e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 1 Nov 2018 11:51:26 +0000 Subject: [PATCH 021/511] Add missing STATIC_ROOT. Thanks @jrvarma. --- etesync_server/settings.py | 1 + 1 file changed, 1 insertion(+) diff --git a/etesync_server/settings.py b/etesync_server/settings.py index 9556167..a8455c2 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -127,6 +127,7 @@ USE_TZ = True # https://docs.djangoproject.com/en/2.0/howto/static-files/ STATIC_URL = '/static/' +STATIC_ROOT = os.path.join(BASE_DIR, 'static/') JOURNAL_API_PERMISSIONS = ( From 3f6dd74e33fbecfeb8dba40570e6dc41477f9e4e Mon Sep 17 00:00:00 2001 From: "Prof. Jayanth R Varma" Date: Wed, 7 Nov 2018 02:18:46 +0530 Subject: [PATCH 022/511] Add example config for using using nginx with uwsgi --- example-configs/nginx-uwsgi/etesync.ini | 15 ++++++++++ .../nginx-uwsgi/my.server.name.conf | 30 +++++++++++++++++++ example-configs/nginx-uwsgi/readme.md | 20 +++++++++++++ example-configs/nginx-uwsgi/uwsgi.service | 15 ++++++++++ 4 files changed, 80 insertions(+) create mode 100644 example-configs/nginx-uwsgi/etesync.ini create mode 100644 example-configs/nginx-uwsgi/my.server.name.conf create mode 100644 example-configs/nginx-uwsgi/readme.md create mode 100644 example-configs/nginx-uwsgi/uwsgi.service diff --git a/example-configs/nginx-uwsgi/etesync.ini b/example-configs/nginx-uwsgi/etesync.ini new file mode 100644 index 0000000..e79eeee --- /dev/null +++ b/example-configs/nginx-uwsgi/etesync.ini @@ -0,0 +1,15 @@ +# uwsgi configuration file +# typical location of this file would be /etc/uwsgi/sites/etesync.ini + +[uwsgi] +socket = /path/to/etesync_server.sock +chown-socket = EtesyncUser:www-data +chmod-socket = 660 +vacuum = true + + +uid = EtesyncUser +chdir = /path/to/etesync +home = %(chdir)/.venv +module = etesync_server.wsgi +master = true diff --git a/example-configs/nginx-uwsgi/my.server.name.conf b/example-configs/nginx-uwsgi/my.server.name.conf new file mode 100644 index 0000000..b5b019d --- /dev/null +++ b/example-configs/nginx-uwsgi/my.server.name.conf @@ -0,0 +1,30 @@ +# nginx configuration for etesync server running on https://my.server.name +# typical location of this file would be /etc/nginx/sites-available/my.server.name.conf + +server { + server_name my.server.name; + + root /srv/http/etesync_server; + + client_max_body_size 5M; + + location /static { + expires 1y; + try_files $uri $uri/ =404; + } + + location / { + uwsgi_pass unix:/path/to/etesync_server.sock; + include uwsgi_params; + } + + # change 443 to say 9443 to run on a non standard port + listen 443 ssl; + listen [::]:443 ssl; + # Enable these two instead of the two above if your nginx supports http2 + # listen 443 ssl http2; + # listen [::]:443 ssl http2; + + ssl_certificate /path/to/certificate-file + ssl_certificate_key /path/to/certificate-key-file + # other ssl directives as needed diff --git a/example-configs/nginx-uwsgi/readme.md b/example-configs/nginx-uwsgi/readme.md new file mode 100644 index 0000000..dad98b6 --- /dev/null +++ b/example-configs/nginx-uwsgi/readme.md @@ -0,0 +1,20 @@ +# Running `etesync` under `nginx` and `uwsgi` + +This configuration assumes that etesync server has been installed in the home folder of a non privileged user +called `EtesyncUser` following the instructions in . Also that static +files have been collected at `/srv/http/etesync_server` by running the following commands: + + sudo mkdir -p /srv/http/etesync_server/static + sudo chown -R EtesyncUser /srv/http/etesync_server + sudo su EtesyncUser + cd /path/to/etesync + ln -s /srv/http/etesync_server/static static + ./manage.py collectstatic + +It is also assumed that `nginx` and `uwsgi` have been installed system wide by `root`, and that `nginx` is running as user/group `www-data`. + +In this setup, `uwsgi` running as a `systemd` service as `root` creates a unix socket with read-write access +to both `EtesyncUser` and `nginx`. It then drops its `root` privilege and runs `etesync` as `EtesyncUser`. + +`nginx` listens on the `https` port (or a non standard port `https` port if desired), delivers static pages directly +and for everything else, communicates with `etesync` over the unix socket. diff --git a/example-configs/nginx-uwsgi/uwsgi.service b/example-configs/nginx-uwsgi/uwsgi.service new file mode 100644 index 0000000..9941ec3 --- /dev/null +++ b/example-configs/nginx-uwsgi/uwsgi.service @@ -0,0 +1,15 @@ +# systemd unit for running uwsgi in emperor mode +# typical location of this file would be /etc/systemd/system/uwsgi.service + +[Unit] +Description=uWSGI Emperor service + +[Service] +ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites +Restart=always +KillSignal=SIGQUIT +Type=notify +NotifyAccess=all + +[Install] +WantedBy=multi-user.target From 9536bed501eae3b99cba39112177d0f4ced57682 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 12 Jan 2019 00:20:22 +0000 Subject: [PATCH 023/511] Update journal manager to the latest version. --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 65d8a96..18e1cba 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,4 +3,4 @@ django-cors-headers==2.3.0 djangorestframework>=3.8.2,<3.8.999 drf-nested-routers==0.90.2 pytz -git+git://github.com/etesync/journal-manager@v0.5.2 +git+git://github.com/etesync/journal-manager@v0.5.4 From aae09ecd65fd7dfe4cbbdab433ae5394eec49e6a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 4 Feb 2019 14:21:41 +0000 Subject: [PATCH 024/511] README: Add etesync logo. --- README.md | 5 ++ icon.svg | 241 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 246 insertions(+) create mode 100644 icon.svg diff --git a/README.md b/README.md index d390022..4725bce 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,8 @@ +

+ +

EteSync - Secure Data Sync

+

+ A skeleton app for running your own [EteSync](https://www.etesync.com) server # Installation diff --git a/icon.svg b/icon.svg new file mode 100644 index 0000000..4827d1d --- /dev/null +++ b/icon.svg @@ -0,0 +1,241 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From d30708f292a4a8c18e7db9a9fc4fffbe3658b853 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 23 Feb 2019 16:31:16 +0000 Subject: [PATCH 025/511] Update requirements. --- requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index 18e1cba..a906399 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ -Django>=2.0.5,<2.0.999 +Django>=2.1.17,<2.0.999 django-cors-headers==2.3.0 +django-etesync-journal==1.0.0 djangorestframework>=3.8.2,<3.8.999 drf-nested-routers==0.90.2 pytz -git+git://github.com/etesync/journal-manager@v0.5.4 From 60f3515b75126dc9144933453525f508f885650a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 23 Feb 2019 18:54:22 +0000 Subject: [PATCH 026/511] Fix django dep. --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index a906399..83d8649 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -Django>=2.1.17,<2.0.999 +Django>=2.1.17,<2.1.999 django-cors-headers==2.3.0 django-etesync-journal==1.0.0 djangorestframework>=3.8.2,<3.8.999 From e531bf2bb974564067132d7010d6a368c4d1dd56 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 23 Feb 2019 19:14:45 +0000 Subject: [PATCH 027/511] Fix django dep yet again. --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 83d8649..35c2f41 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -Django>=2.1.17,<2.1.999 +Django>=2.1.7,<2.1.999 django-cors-headers==2.3.0 django-etesync-journal==1.0.0 djangorestframework>=3.8.2,<3.8.999 From b9de109998dc907b99ca84b410ca3c6f0c2bffbf Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 8 Mar 2019 16:41:37 +0000 Subject: [PATCH 028/511] README: improve the update instructions Add the missing call to migrate the db. Reorganise them to make them clearer. --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4725bce..fd41218 100644 --- a/README.md +++ b/README.md @@ -82,10 +82,12 @@ for more information about the `SECRET_KEY` and its uses. # Updating -Inside the virtualenv, run `pip install -U -r requirements.txt` to update -dependencies to latest compatible versions of Django and -djangorestframework (it will only update to latest patch level which should -be API-compatible). +First, run `git pull --rebase` to update this repository. +Then, inside the virtualenv: +1. Run `pip install -U -r requirements.txt` to update the dependencies. +2. Run `python manage.py migrate` to perform database migrations. + +You can now restart the server. # Supporting EteSync From e8022a0986fa4bc3085f4764c5d5256af33b66d1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 15 Mar 2019 08:39:14 +0000 Subject: [PATCH 029/511] README: Improve installation instructions for Ubuntu As discussed in #12 --- README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index fd41218..e1f6608 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,12 @@ A skeleton app for running your own [EteSync](https://www.etesync.com) server # Installation -To setup your own EteSync server based on the git version just clone this -git repository and set up the django app: +Before installing the EteSync server make sure you install `virtualenv` (for **Python 3**): + +* Arch Linux: `pacman -S python-virtualenv` +* Debian/Ubuntu: `apt-get install python3-virtualenv` + +Then just clone the git repo and set up this app: ``` git clone https://github.com/etesync/server-skeleton.git @@ -16,8 +20,8 @@ git clone https://github.com/etesync/server-skeleton.git cd server-skeleton # Set up the environment and deps -virtualenv3 .venv -source .venv/bin/activate +virtualenv -p python3 venv # If doesn't work, try: virtualenv3 venv +source venv/bin/activate pip install -r requirements.txt ``` From ac2428af88df4ccf493842187e5f4694d23485db Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 25 Apr 2019 17:14:24 +0100 Subject: [PATCH 030/511] Bump etesync journal requirement. --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 35c2f41..b0703c2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ Django>=2.1.7,<2.1.999 django-cors-headers==2.3.0 -django-etesync-journal==1.0.0 +django-etesync-journal==1.0.1 djangorestframework>=3.8.2,<3.8.999 drf-nested-routers==0.90.2 pytz From c1eec9e7968ffb9e860dc151b92f72eaf247c272 Mon Sep 17 00:00:00 2001 From: Pierre-Alain TORET Date: Mon, 29 Apr 2019 16:42:13 +0200 Subject: [PATCH 031/511] README.md : add production deployment section --- README.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e1f6608..de37b3d 100644 --- a/README.md +++ b/README.md @@ -47,15 +47,24 @@ Now you can initialise our django app ./manage.py migrate ``` -And you are done! You can now either run the debug server just to see everything works as expected by running: +And you are done! You can now run the debug server just to see everything works as expected by running: ``` ./manage.py runserver 0.0.0.0:8000 ``` -Using the debug server in production is not recommended, so you should configure your webserver to serve -etesync (with TLS). An example on how to do so with nginx can be found [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). +Using the debug server in production is not recommended, so please read the following section for a proper deployment. +# Production deployment + +EteSync is based on Django so you should : + * either follow the instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). + * either/or follow the instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). + +The webserver should also be configured to serve EteSync using TLS. + +There are more details about a proper production setup in the [wiki](https://github.com/etesync/server-skeleton/wiki). + # Usage Create yourself an admin user: From ac474d669aeb23924106784b7920d15d634c6a54 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 1 May 2019 10:54:24 +0100 Subject: [PATCH 032/511] Bump etesync journal requirement to 1.0.2 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index b0703c2..23b8d30 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ Django>=2.1.7,<2.1.999 django-cors-headers==2.3.0 -django-etesync-journal==1.0.1 +django-etesync-journal==1.0.2 djangorestframework>=3.8.2,<3.8.999 drf-nested-routers==0.90.2 pytz From 3d4321e50c449a646dc05c4adc7014665559329f Mon Sep 17 00:00:00 2001 From: Pierre-Alain TORET Date: Wed, 1 May 2019 12:40:48 +0200 Subject: [PATCH 033/511] Support loading configuration from an external file --- README.md | 12 ++++++----- etesync-server.ini | 10 ++++++++++ etesync_server/settings.py | 41 +++++++++++++++++++++++++++----------- 3 files changed, 46 insertions(+), 17 deletions(-) create mode 100644 etesync-server.ini diff --git a/README.md b/README.md index de37b3d..36a3262 100644 --- a/README.md +++ b/README.md @@ -26,11 +26,13 @@ source venv/bin/activate pip install -r requirements.txt ``` -Edit the [settings file](etesync_server/settings.py). Please refer to the -[Django deployment -checklist](https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/) -for full instructions on how to configure a Django app for production. Some -particular settings that should be edited are: +If you are familiar with Django you can just edit the [settings file](etesync_server/settings.py) +according to the [Django deployment checklist](https://docs.djangoproject.com/en/1.11/howto/deployment/checklist) +if you are not, we also provide a simple [configuration file](etesync-server.ini) +for easy deployment which you can use. You can either edit the provided file or +create one in `/etc/etesync-server`. + +Some particular settings that should be edited are: * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) -- this is the list of host/domain names or addresses on which the app will be served diff --git a/etesync-server.ini b/etesync-server.ini new file mode 100644 index 0000000..0dabba0 --- /dev/null +++ b/etesync-server.ini @@ -0,0 +1,10 @@ +[global] +secret_file = secret.txt +debug = false + +[allowed_hosts] +;allowed_host1 = host1.tld + +[database] +engine = django.db.backends.sqlite3 +name = db.sqlite3 diff --git a/etesync_server/settings.py b/etesync_server/settings.py index a8455c2..2df2914 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -11,6 +11,7 @@ https://docs.djangoproject.com/en/2.0/ref/settings/ """ import os +import configparser from .utils import get_secret_from_file # Build paths inside the project like this: os.path.join(BASE_DIR, ...) @@ -30,6 +31,34 @@ DEBUG = False ALLOWED_HOSTS = [] +# Database +# https://docs.djangoproject.com/en/2.0/ref/settings/#databases + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.environ.get('ETESYNC_DB_PATH', + os.path.join(BASE_DIR, 'db.sqlite3')), + } +} + + + +# Define where to find configuration files +config_locations = ['etesync-server.ini', '/etc/etesync-server/etesync-server.ini'] +# Use config file if present +if any(os.path.isfile(x) for x in config_locations): + config = configparser.ConfigParser() + config.read(config_locations) + + SECRET_FILE = config['global']['secret_file'] + + DEBUG = config['global'].getboolean('debug') + + ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')] + + DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } } + # Application definition @@ -78,18 +107,6 @@ TEMPLATES = [ WSGI_APPLICATION = 'etesync_server.wsgi.application' -# Database -# https://docs.djangoproject.com/en/2.0/ref/settings/#databases - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': os.environ.get('ETESYNC_DB_PATH', - os.path.join(BASE_DIR, 'db.sqlite3')), - } -} - - # Password validation # https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators From 654b3fe860273a1deddaab3ff59c2aabac090b8a Mon Sep 17 00:00:00 2001 From: Pierre-Alain TORET Date: Tue, 21 May 2019 13:00:13 +0200 Subject: [PATCH 034/511] README.md : add instructions for installing from packages --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 36a3262..258c50f 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,13 @@ A skeleton app for running your own [EteSync](https://www.etesync.com) server # Installation +## Using pre-built packages + +* Arch Linux : [AUR](https://aur.archlinux.org/packages/etesync-server) +* Fedora : [COPR](https://copr.fedorainfracloud.org/coprs/daftaupe/etesync) + +## Manually from source + Before installing the EteSync server make sure you install `virtualenv` (for **Python 3**): * Arch Linux: `pacman -S python-virtualenv` @@ -26,6 +33,8 @@ source venv/bin/activate pip install -r requirements.txt ``` +# Configuration + If you are familiar with Django you can just edit the [settings file](etesync_server/settings.py) according to the [Django deployment checklist](https://docs.djangoproject.com/en/1.11/howto/deployment/checklist) if you are not, we also provide a simple [configuration file](etesync-server.ini) From a87a4deec18f5c6954153c6e531243cf89ec7a4b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 6 Jun 2019 13:11:03 +0100 Subject: [PATCH 035/511] Deps: update drf. --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 23b8d30..d2077a6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ Django>=2.1.7,<2.1.999 django-cors-headers==2.3.0 django-etesync-journal==1.0.2 -djangorestframework>=3.8.2,<3.8.999 +djangorestframework>=3.9.4,<3.9.999 drf-nested-routers==0.90.2 pytz From 3ccb40055e3b721a70406db87bb5bc1e09e7141d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 7 Jun 2019 08:53:50 +0100 Subject: [PATCH 036/511] Add an index page to make it more obvious it works. People were getting confused by the index page returning a 404, so this commit adds an index page so it returns a 200 and looks like it works. --- etesync_server/settings.py | 4 +++- etesync_server/urls.py | 2 ++ templates/success.html | 12 ++++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) create mode 100644 templates/success.html diff --git a/etesync_server/settings.py b/etesync_server/settings.py index 2df2914..cfe339f 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -91,7 +91,9 @@ ROOT_URLCONF = 'etesync_server.urls' TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', - 'DIRS': [], + 'DIRS': [ + os.path.join(BASE_DIR, 'templates') + ], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ diff --git a/etesync_server/urls.py b/etesync_server/urls.py index c76d88e..8b8fc53 100644 --- a/etesync_server/urls.py +++ b/etesync_server/urls.py @@ -15,6 +15,7 @@ Including another URLconf """ from django.urls import include, path, re_path from django.contrib import admin +from django.views.generic import TemplateView from rest_framework_nested import routers from rest_framework.authtoken import views as token_views @@ -37,4 +38,5 @@ urlpatterns = [ re_path(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), re_path(r'^api-token-auth/', token_views.obtain_auth_token), path('admin/', admin.site.urls), + path('', TemplateView.as_view(template_name='success.html')), ] diff --git a/templates/success.html b/templates/success.html new file mode 100644 index 0000000..c7cf494 --- /dev/null +++ b/templates/success.html @@ -0,0 +1,12 @@ + + + + It works! + + +

It works!

+

+ Please refer to the README to complete the final steps if you haven't done so already. +

+ + From 48390b0b01cc701830eabeea8ef0988375bd6651 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 2 Aug 2019 19:39:59 +0100 Subject: [PATCH 037/511] README: add a note about the example configs As suggested in #5 --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 258c50f..47f6fdb 100644 --- a/README.md +++ b/README.md @@ -68,9 +68,10 @@ Using the debug server in production is not recommended, so please read the foll # Production deployment -EteSync is based on Django so you should : - * either follow the instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). - * either/or follow the instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). +EteSync is based on Django so you should refer to one of the following + * The instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). + * Tnstructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). + * The [example configurations](example-configs) in this repo. The webserver should also be configured to serve EteSync using TLS. From c3b50d3f72a383b265a2922898e252f3eb9af604 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 4 Aug 2019 16:11:47 +0100 Subject: [PATCH 038/511] Fix issue with etesync-server.ini always being loaded. Fixes #20. --- README.md | 9 +++++---- etesync-server.ini => etesync-server.ini.example | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) rename etesync-server.ini => etesync-server.ini.example (82%) diff --git a/README.md b/README.md index 47f6fdb..99005c4 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,11 @@ pip install -r requirements.txt # Configuration If you are familiar with Django you can just edit the [settings file](etesync_server/settings.py) -according to the [Django deployment checklist](https://docs.djangoproject.com/en/1.11/howto/deployment/checklist) -if you are not, we also provide a simple [configuration file](etesync-server.ini) -for easy deployment which you can use. You can either edit the provided file or -create one in `/etc/etesync-server`. +according to the [Django deployment checklist](https://docs.djangoproject.com/en/dev/howto/deployment/checklist/) +if you are not, we also provide a simple [configuration file](etesync-server.ini.example) +for easy deployment which you can use. + +To use the easy configuration file rename it to `example-server.ini` and place it either at the root of this repository or in `/etc/etesync-server`. Some particular settings that should be edited are: * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) diff --git a/etesync-server.ini b/etesync-server.ini.example similarity index 82% rename from etesync-server.ini rename to etesync-server.ini.example index 0dabba0..3b8b518 100644 --- a/etesync-server.ini +++ b/etesync-server.ini.example @@ -3,7 +3,7 @@ secret_file = secret.txt debug = false [allowed_hosts] -;allowed_host1 = host1.tld +allowed_host1 = example.com [database] engine = django.db.backends.sqlite3 From cf0dc8e6a81f0b87bfba6d952ce9085d21257c18 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 17 Sep 2019 10:14:00 +0100 Subject: [PATCH 039/511] README: make it clearer that it's not only for Arch/Fedora --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 99005c4..ee4df43 100644 --- a/README.md +++ b/README.md @@ -12,12 +12,13 @@ A skeleton app for running your own [EteSync](https://www.etesync.com) server * Arch Linux : [AUR](https://aur.archlinux.org/packages/etesync-server) * Fedora : [COPR](https://copr.fedorainfracloud.org/coprs/daftaupe/etesync) -## Manually from source +## From source Before installing the EteSync server make sure you install `virtualenv` (for **Python 3**): * Arch Linux: `pacman -S python-virtualenv` * Debian/Ubuntu: `apt-get install python3-virtualenv` +* Mac/Windows/Other Linux: install virtualenv or just skip the instructions mentioning virtualenv. Then just clone the git repo and set up this app: From 61ae366a7b41677d5847bbe3b18b4c8edd18630f Mon Sep 17 00:00:00 2001 From: StefanAbl <48151667+StefanAbl@users.noreply.github.com> Date: Wed, 2 Oct 2019 22:42:37 +0200 Subject: [PATCH 040/511] added missing closing bracket --- example-configs/nginx-uwsgi/my.server.name.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/example-configs/nginx-uwsgi/my.server.name.conf b/example-configs/nginx-uwsgi/my.server.name.conf index b5b019d..53b4fb7 100644 --- a/example-configs/nginx-uwsgi/my.server.name.conf +++ b/example-configs/nginx-uwsgi/my.server.name.conf @@ -28,3 +28,4 @@ server { ssl_certificate /path/to/certificate-file ssl_certificate_key /path/to/certificate-key-file # other ssl directives as needed +} From 94ae4eaa02998ab6f61eb58e17eddf83120341d2 Mon Sep 17 00:00:00 2001 From: edleeman Date: Mon, 7 Oct 2019 20:09:09 +0100 Subject: [PATCH 041/511] Update README Updated README to provide the correct example template name. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ee4df43..234ac0b 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ according to the [Django deployment checklist](https://docs.djangoproject.com/en if you are not, we also provide a simple [configuration file](etesync-server.ini.example) for easy deployment which you can use. -To use the easy configuration file rename it to `example-server.ini` and place it either at the root of this repository or in `/etc/etesync-server`. +To use the easy configuration file rename it to `etesync-server.ini` and place it either at the root of this repository or in `/etc/etesync-server`. Some particular settings that should be edited are: * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) From de71aaa47641260caf6b4a2ac61f418c60611dca Mon Sep 17 00:00:00 2001 From: kumy Date: Tue, 22 Oct 2019 23:42:13 +0200 Subject: [PATCH 042/511] Fix typo in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 234ac0b..1761186 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ Using the debug server in production is not recommended, so please read the foll EteSync is based on Django so you should refer to one of the following * The instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). - * Tnstructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). + * Instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). * The [example configurations](example-configs) in this repo. The webserver should also be configured to serve EteSync using TLS. From c1f790cad148e46e90f5bb925f7691156272c8fd Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 21 Jan 2020 09:19:08 +0200 Subject: [PATCH 043/511] Update dependencies. --- requirements.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/requirements.txt b/requirements.txt index d2077a6..2c0e540 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ Django>=2.1.7,<2.1.999 -django-cors-headers==2.3.0 -django-etesync-journal==1.0.2 -djangorestframework>=3.9.4,<3.9.999 -drf-nested-routers==0.90.2 +django-cors-headers==3.2.1 +django-etesync-journal==1.1.0 +djangorestframework>=3.11.0,<3.11.999 +drf-nested-routers==0.91 pytz From b026643cceae07b039942bf0c990ccf917eb072a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 21 Jan 2020 09:20:18 +0200 Subject: [PATCH 044/511] Update code according to drf changes. --- etesync_server/urls.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etesync_server/urls.py b/etesync_server/urls.py index 8b8fc53..3e4e12c 100644 --- a/etesync_server/urls.py +++ b/etesync_server/urls.py @@ -28,8 +28,8 @@ router.register(r'journal/(?P[^/]+)', views.EntryViewSet) router.register(r'user', views.UserInfoViewSet) journals_router = routers.NestedSimpleRouter(router, r'journals', lookup='journal') -journals_router.register(r'members', views.MembersViewSet, base_name='journal-members') -journals_router.register(r'entries', views.EntryViewSet, base_name='journal-entries') +journals_router.register(r'members', views.MembersViewSet, basename='journal-members') +journals_router.register(r'entries', views.EntryViewSet, basename='journal-entries') urlpatterns = [ From 22198d387d5a7ee354308b83e5315aa033864772 Mon Sep 17 00:00:00 2001 From: Pierre-Alain TORET Date: Tue, 5 Nov 2019 10:30:50 +0100 Subject: [PATCH 045/511] Make settings in .ini optional and add new ones Fixes #24 New settings available : STATIC_ROOT STATIC_URL LANGUAGE_CODE TIME_ZONE --- etesync-server.ini.example | 5 +++++ etesync_server/settings.py | 39 +++++++++++++++++++++----------------- 2 files changed, 27 insertions(+), 17 deletions(-) diff --git a/etesync-server.ini.example b/etesync-server.ini.example index 3b8b518..424a26d 100644 --- a/etesync-server.ini.example +++ b/etesync-server.ini.example @@ -1,6 +1,11 @@ [global] secret_file = secret.txt debug = false +;Advanced options, only uncomment if you know what you're doing: +;static_root = /path/to/static +;static_url = /static/ +;language_code = en-us +;time_zone = UTC [allowed_hosts] allowed_host1 = example.com diff --git a/etesync_server/settings.py b/etesync_server/settings.py index cfe339f..ab7ff8b 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -43,23 +43,6 @@ DATABASES = { } - -# Define where to find configuration files -config_locations = ['etesync-server.ini', '/etc/etesync-server/etesync-server.ini'] -# Use config file if present -if any(os.path.isfile(x) for x in config_locations): - config = configparser.ConfigParser() - config.read(config_locations) - - SECRET_FILE = config['global']['secret_file'] - - DEBUG = config['global'].getboolean('debug') - - ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')] - - DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } } - - # Application definition INSTALLED_APPS = [ @@ -149,6 +132,28 @@ STATIC_URL = '/static/' STATIC_ROOT = os.path.join(BASE_DIR, 'static/') +# Define where to find configuration files +config_locations = ['etesync-server.ini', '/etc/etesync-server/etesync-server.ini'] +# Use config file if present +if any(os.path.isfile(x) for x in config_locations): + config = configparser.ConfigParser() + config.read(config_locations) + + section = config['global'] + + SECRET_FILE = section.get('secret_file', SECRET_FILE) + STATIC_ROOT = section.get('static_root', STATIC_ROOT) + STATIC_URL = section.get('static_url', STATIC_URL) + LANGUAGE_CODE = section.get('language_code', LANGUAGE_CODE) + TIME_ZONE = section.get('time_zone', TIME_ZONE) + DEBUG = section.getboolean('debug', DEBUG) + + if 'allowed_hosts' in config: + ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')] + + if 'database' in config: + DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } } + JOURNAL_API_PERMISSIONS = ( 'rest_framework.permissions.IsAuthenticated', ) From 8c71bcddcab016ba7c086176a7354d1278f12bff Mon Sep 17 00:00:00 2001 From: Pierre-Alain TORET Date: Thu, 30 Jan 2020 10:51:25 +0100 Subject: [PATCH 046/511] Update etesync journal requirement --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 2c0e540..3c87833 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,6 @@ Django>=2.1.7,<2.1.999 django-cors-headers==3.2.1 -django-etesync-journal==1.1.0 +django-etesync-journal==1.2.0 djangorestframework>=3.11.0,<3.11.999 drf-nested-routers==0.91 pytz From 1a04d8ec6a0852a058c3f87b58b06698f2c5e164 Mon Sep 17 00:00:00 2001 From: Simon Vandevelde Date: Mon, 3 Feb 2020 20:11:16 +0100 Subject: [PATCH 047/511] README: Added links pointing to specific wikipages * Added links pointing to the wiki. I added a link for the deployment setup and one for the TLS setup. Fixes #35 --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1761186..33aaf64 100644 --- a/README.md +++ b/README.md @@ -75,9 +75,10 @@ EteSync is based on Django so you should refer to one of the following * Instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). * The [example configurations](example-configs) in this repo. -The webserver should also be configured to serve EteSync using TLS. +There are more details about a proper production setup using uWSGI and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-uWSGI-and-Nginx). -There are more details about a proper production setup in the [wiki](https://github.com/etesync/server-skeleton/wiki). +The webserver should also be configured to serve EteSync using TLS. +A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-EteSync) as well. # Usage From 8fdaccdc4eb709dd32b33ace427d6983613598d7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 14 Feb 2020 16:20:33 +0200 Subject: [PATCH 048/511] Provide more explicit copyright and licensing information. Fixes #37 --- etesync_server/__init__.py | 14 ++++++++++++++ etesync_server/settings.py | 14 ++++++++++++++ etesync_server/urls.py | 14 ++++++++++++++ etesync_server/utils.py | 14 ++++++++++++++ 4 files changed, 56 insertions(+) diff --git a/etesync_server/__init__.py b/etesync_server/__init__.py index e69de29..227e8c9 100644 --- a/etesync_server/__init__.py +++ b/etesync_server/__init__.py @@ -0,0 +1,14 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + diff --git a/etesync_server/settings.py b/etesync_server/settings.py index ab7ff8b..cca7915 100644 --- a/etesync_server/settings.py +++ b/etesync_server/settings.py @@ -1,3 +1,17 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + """ Django settings for etesync_server project. diff --git a/etesync_server/urls.py b/etesync_server/urls.py index 3e4e12c..5f3a6d7 100644 --- a/etesync_server/urls.py +++ b/etesync_server/urls.py @@ -1,3 +1,17 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + """etesync_server URL Configuration The `urlpatterns` list routes URLs to views. For more information please see: diff --git a/etesync_server/utils.py b/etesync_server/utils.py index 8f85f10..21c99f2 100644 --- a/etesync_server/utils.py +++ b/etesync_server/utils.py @@ -1,3 +1,17 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + from django.core.management import utils def get_secret_from_file(path): From 2ac7ec250f15187064d552d1deae6534df6371d0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 14 Feb 2020 16:22:28 +0200 Subject: [PATCH 049/511] Requirements: update django version. --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 3c87833..4195dc6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -Django>=2.1.7,<2.1.999 +Django>=2.2.9,<2.2.999 django-cors-headers==3.2.1 django-etesync-journal==1.2.0 djangorestframework>=3.11.0,<3.11.999 From 228522d0191f4718dfedc5fd3576c02b8a5b3e0e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 19 Feb 2020 14:54:35 +0200 Subject: [PATCH 050/511] Add requirements. --- requirements.in/base.txt | 11 +++++++++++ requirements.in/development.txt | 2 ++ requirements.txt | 30 ++++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+) create mode 100644 requirements.in/base.txt create mode 100644 requirements.in/development.txt create mode 100644 requirements.txt diff --git a/requirements.in/base.txt b/requirements.in/base.txt new file mode 100644 index 0000000..a7d1734 --- /dev/null +++ b/requirements.in/base.txt @@ -0,0 +1,11 @@ +django +django-allauth +django-anymail +django-appconf +django-cors-headers +django-debug-toolbar +django-fullurl +django-ipware +djangorestframework +drf-nested-routers +psycopg2-binary diff --git a/requirements.in/development.txt b/requirements.in/development.txt new file mode 100644 index 0000000..30fb558 --- /dev/null +++ b/requirements.in/development.txt @@ -0,0 +1,2 @@ +coverage +pip-tools diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..34ca428 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,30 @@ +# +# This file is autogenerated by pip-compile +# To update, run: +# +# pip-compile --output-file=requirements.txt requirements.in/base.txt +# +asgiref==3.2.3 # via django +certifi==2019.11.28 # via requests +chardet==3.0.4 # via requests +defusedxml==0.6.0 # via python3-openid +django-allauth==0.41.0 +django-anymail==7.0.0 +django-appconf==1.0.3 +django-cors-headers==3.2.1 +django-debug-toolbar==2.2 +django-fullurl==1.0 +django-ipware==2.1.0 +django==3.0.3 +djangorestframework==3.11.0 +drf-nested-routers==0.91 +idna==2.8 # via requests +oauthlib==3.1.0 # via requests-oauthlib +psycopg2-binary==2.8.4 +python3-openid==3.1.0 # via django-allauth +pytz==2019.3 # via django +requests-oauthlib==1.3.0 # via django-allauth +requests==2.22.0 # via django-allauth, django-anymail, requests-oauthlib +six==1.14.0 # via django-anymail, django-appconf +sqlparse==0.3.0 # via django, django-debug-toolbar +urllib3==1.25.8 # via requests From 703a5ae36aaaeea69885b716a1072c239b69014d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 19 Feb 2020 14:55:56 +0200 Subject: [PATCH 051/511] Create new django project. --- django_etesync/__init__.py | 0 django_etesync/admin.py | 3 +++ django_etesync/apps.py | 5 +++++ django_etesync/migrations/__init__.py | 0 django_etesync/models.py | 3 +++ django_etesync/tests.py | 3 +++ django_etesync/views.py | 3 +++ 7 files changed, 17 insertions(+) create mode 100644 django_etesync/__init__.py create mode 100644 django_etesync/admin.py create mode 100644 django_etesync/apps.py create mode 100644 django_etesync/migrations/__init__.py create mode 100644 django_etesync/models.py create mode 100644 django_etesync/tests.py create mode 100644 django_etesync/views.py diff --git a/django_etesync/__init__.py b/django_etesync/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etesync/admin.py b/django_etesync/admin.py new file mode 100644 index 0000000..8c38f3f --- /dev/null +++ b/django_etesync/admin.py @@ -0,0 +1,3 @@ +from django.contrib import admin + +# Register your models here. diff --git a/django_etesync/apps.py b/django_etesync/apps.py new file mode 100644 index 0000000..adb8f96 --- /dev/null +++ b/django_etesync/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class DjangoEtesyncConfig(AppConfig): + name = 'django_etesync' diff --git a/django_etesync/migrations/__init__.py b/django_etesync/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etesync/models.py b/django_etesync/models.py new file mode 100644 index 0000000..71a8362 --- /dev/null +++ b/django_etesync/models.py @@ -0,0 +1,3 @@ +from django.db import models + +# Create your models here. diff --git a/django_etesync/tests.py b/django_etesync/tests.py new file mode 100644 index 0000000..7ce503c --- /dev/null +++ b/django_etesync/tests.py @@ -0,0 +1,3 @@ +from django.test import TestCase + +# Create your tests here. diff --git a/django_etesync/views.py b/django_etesync/views.py new file mode 100644 index 0000000..91ea44a --- /dev/null +++ b/django_etesync/views.py @@ -0,0 +1,3 @@ +from django.shortcuts import render + +# Create your views here. From 818bb8d70f5d576f0a0d756ba8688527a2ff3dd4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 19 Feb 2020 20:53:43 +0200 Subject: [PATCH 052/511] Create the django_etesync app. --- django_etesync/__init__.py | 1 + django_etesync/app_settings.py | 52 ++++++++ django_etesync/migrations/0001_initial.py | 67 +++++++++++ django_etesync/models.py | 77 +++++++++++- django_etesync/paginators.py | 36 ++++++ django_etesync/permissions.py | 65 ++++++++++ django_etesync/serializers.py | 88 ++++++++++++++ django_etesync/views.py | 140 +++++++++++++++++++++- 8 files changed, 523 insertions(+), 3 deletions(-) create mode 100644 django_etesync/app_settings.py create mode 100644 django_etesync/migrations/0001_initial.py create mode 100644 django_etesync/paginators.py create mode 100644 django_etesync/permissions.py create mode 100644 django_etesync/serializers.py diff --git a/django_etesync/__init__.py b/django_etesync/__init__.py index e69de29..426fefd 100644 --- a/django_etesync/__init__.py +++ b/django_etesync/__init__.py @@ -0,0 +1 @@ +from .app_settings import app_settings diff --git a/django_etesync/app_settings.py b/django_etesync/app_settings.py new file mode 100644 index 0000000..6a04b4e --- /dev/null +++ b/django_etesync/app_settings.py @@ -0,0 +1,52 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +import sys # noqa + + +class AppSettings: + def __init__(self, prefix): + self.prefix = prefix + + def import_from_str(self, name): + from importlib import import_module + + p, m = name.rsplit('.', 1) + + mod = import_module(p) + return getattr(mod, m) + + def _setting(self, name, dflt): + from django.conf import settings + return getattr(settings, self.prefix + name, dflt) + + @property + def API_PERMISSIONS(self): + perms = self._setting("API_PERMISSIONS", ('rest_framework.permissions.IsAuthenticated', )) + ret = [] + for perm in perms: + ret.append(self.import_from_str(perm)) + return ret + + @property + def API_AUTHENTICATORS(self): + perms = self._setting("API_AUTHENTICATORS", ('rest_framework.authentication.TokenAuthentication', + 'rest_framework.authentication.SessionAuthentication')) + ret = [] + for perm in perms: + ret.append(self.import_from_str(perm)) + return ret + + +app_settings = AppSettings('ETESYNC_') diff --git a/django_etesync/migrations/0001_initial.py b/django_etesync/migrations/0001_initial.py new file mode 100644 index 0000000..6fa724b --- /dev/null +++ b/django_etesync/migrations/0001_initial.py @@ -0,0 +1,67 @@ +# Generated by Django 3.0.3 on 2020-02-19 15:33 + +from django.conf import settings +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.CreateModel( + name='Collection', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-fA-F0-9\\-_=]{44}')])), + ('version', models.PositiveSmallIntegerField()), + ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ], + options={ + 'unique_together': {('uid', 'owner')}, + }, + ), + migrations.CreateModel( + name='CollectionItem', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-fA-F0-9\\-_=]{44}')])), + ('version', models.PositiveSmallIntegerField()), + ('encryptionKey', models.BinaryField(editable=True)), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etesync.Collection')), + ], + options={ + 'unique_together': {('uid', 'collection')}, + }, + ), + migrations.CreateModel( + name='CollectionItemSnapshot', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('current', models.BooleanField(default=True)), + ('chunkHmac', models.CharField(max_length=50)), + ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etesync.CollectionItem')), + ], + options={ + 'unique_together': {('item', 'current')}, + }, + ), + migrations.CreateModel( + name='CollectionItemChunk', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-fA-F0-9\\-_=]{44}')])), + ('order', models.CharField(max_length=100)), + ('itemSnapshot', models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, to='django_etesync.CollectionItemSnapshot')), + ], + options={ + 'ordering': ['order'], + }, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 71a8362..9e578ba 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -1,3 +1,78 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + from django.db import models +from django.conf import settings +from django.core.validators import RegexValidator +from django.utils.functional import cached_property + + +UidValidator = RegexValidator(regex=r'[a-zA-Z0-9\-_=]{44}', message='Not a valid UID. Expected a 256bit base64url.') + + +class Collection(models.Model): + uid = models.CharField(db_index=True, blank=False, null=False, + max_length=44, validators=[UidValidator]) + version = models.PositiveSmallIntegerField() + owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + + class Meta: + unique_together = ('uid', 'owner') + + def __str__(self): + return self.uid + + +class CollectionItem(models.Model): + uid = models.CharField(db_index=True, blank=False, null=False, + max_length=44, validators=[UidValidator]) + version = models.PositiveSmallIntegerField() + encryptionKey = models.BinaryField(editable=True, blank=False, null=False) + collection = models.ForeignKey(Collection, on_delete=models.CASCADE) + + class Meta: + unique_together = ('uid', 'collection') + + @cached_property + def content(self): + return self.snapshots.get(current=True) + + def __str__(self): + return self.uid + + +class CollectionItemSnapshot(models.Model): + item = models.ForeignKey(CollectionItem, related_name='snapshots', on_delete=models.CASCADE) + current = models.BooleanField(default=True) + chunkHmac = models.CharField(max_length=50, blank=False, null=False) + + class Meta: + unique_together = ('item', 'current') + + def __str__(self): + return "{}, current={}".format(self.item.uid, self.current) + + +class CollectionItemChunk(models.Model): + uid = models.CharField(db_index=True, blank=False, null=False, + max_length=44, validators=[UidValidator]) + itemSnapshot = models.ForeignKey(CollectionItemSnapshot, related_name='chunks', null=True, on_delete=models.SET_NULL) + order = models.CharField(max_length=100, blank=False, null=False) + + class Meta: + # unique_together = ('itemSnapshot', 'order') # Currently off because we set the item snapshot to null on deletion + ordering = ['order'] -# Create your models here. + def __str__(self): + return self.uid diff --git a/django_etesync/paginators.py b/django_etesync/paginators.py new file mode 100644 index 0000000..6d55599 --- /dev/null +++ b/django_etesync/paginators.py @@ -0,0 +1,36 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +from rest_framework import pagination +from rest_framework.response import Response + + +class LinkHeaderPagination(pagination.LimitOffsetPagination): + def get_paginated_response(self, data): + next_url = self.get_next_link() + previous_url = self.get_previous_link() + + if next_url is not None and previous_url is not None: + link = '<{next_url}>; rel="next", <{previous_url}>; rel="prev"' + elif next_url is not None: + link = '<{next_url}>; rel="next"' + elif previous_url is not None: + link = '<{previous_url}>; rel="prev"' + else: + link = '' + + link = link.format(next_url=next_url, previous_url=previous_url) + headers = {'Link': link} if link else {} + + return Response(data, headers=headers) diff --git a/django_etesync/permissions.py b/django_etesync/permissions.py new file mode 100644 index 0000000..f553930 --- /dev/null +++ b/django_etesync/permissions.py @@ -0,0 +1,65 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +from rest_framework import permissions +from journal.models import Journal, JournalMember + + +class IsOwnerOrReadOnly(permissions.BasePermission): + """ + Custom permission to only allow owners of an object to edit it. + """ + + def has_object_permission(self, request, view, obj): + if request.method in permissions.SAFE_METHODS: + return True + + return obj.owner == request.user + + +class IsJournalOwner(permissions.BasePermission): + """ + Custom permission to only allow owners of a journal to view it + """ + + def has_permission(self, request, view): + journal_uid = view.kwargs['journal_uid'] + try: + journal = view.get_journal_queryset().get(uid=journal_uid) + return journal.owner == request.user + except Journal.DoesNotExist: + # If the journal does not exist, we want to 404 later, not permission denied. + return True + + +class IsMemberReadOnly(permissions.BasePermission): + """ + Custom permission to make a journal read only if a read only member + """ + + def has_permission(self, request, view): + if request.method in permissions.SAFE_METHODS: + return True + + journal_uid = view.kwargs['journal_uid'] + try: + journal = view.get_journal_queryset().get(uid=journal_uid) + member = journal.members.get(user=request.user) + return not member.readOnly + except Journal.DoesNotExist: + # If the journal does not exist, we want to 404 later, not permission denied. + return True + except JournalMember.DoesNotExist: + # Not being a member means we are the owner. + return True diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py new file mode 100644 index 0000000..d4f259f --- /dev/null +++ b/django_etesync/serializers.py @@ -0,0 +1,88 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +import base64 + +from django.contrib.auth import get_user_model +from rest_framework import serializers +from . import models + +User = get_user_model() + + +class BinaryBase64Field(serializers.Field): + def to_representation(self, value): + return base64.b64encode(value).decode('ascii') + + def to_internal_value(self, data): + return base64.b64decode(data) + + +class CollectionSerializer(serializers.ModelSerializer): + owner = serializers.SlugRelatedField( + slug_field=User.USERNAME_FIELD, + read_only=True + ) + encryptionKey = serializers.SerializerMethodField('get_key_from_context') + permissions = serializers.SerializerMethodField('get_permission_from_context') + ctag = serializers.SerializerMethodField('get_ctag') + + class Meta: + model = models.Collection + fields = ('uid', 'version', 'owner', 'encryptionKey', 'permissions', 'ctag') + + def get_key_from_context(self, obj): + request = self.context.get('request', None) + if request is not None: + return 'FIXME' + return None + + def get_permission_from_context(self, obj): + request = self.context.get('request', None) + if request is not None: + return 'FIXME' + return 'readOnly' + + def get_ctag(self, obj): + return 'FIXME' + + +class CollectionItemChunkSerializer(serializers.ModelSerializer): + class Meta: + model = models.CollectionItemChunk + fields = ('uid', ) + + +class CollectionItemSnapshotSerializer(serializers.ModelSerializer): + chunks = serializers.SlugRelatedField( + slug_field='uid', + queryset=models.CollectionItemChunk, + many=True + ) + + class Meta: + model = models.CollectionItemSnapshot + fields = ('chunks', 'chunkHmac') + + +class CollectionItemSerializer(serializers.ModelSerializer): + encryptionKey = BinaryBase64Field() + content = CollectionItemSnapshotSerializer( + read_only=True, + many=False + ) + + class Meta: + model = models.CollectionItem + fields = ('uid', 'version', 'encryptionKey', 'content') diff --git a/django_etesync/views.py b/django_etesync/views.py index 91ea44a..4d99dad 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -1,3 +1,139 @@ -from django.shortcuts import render +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . -# Create your views here. +from django.conf import settings +from django.contrib.auth import login, get_user_model +from django.db import IntegrityError, transaction +from django.db.models import Q +from django.core.exceptions import ObjectDoesNotExist +from django.http import HttpResponseBadRequest, HttpResponse, Http404 +from django.shortcuts import get_object_or_404 +from django.views.decorators.csrf import csrf_exempt +from django.views.decorators.http import require_POST + +from rest_framework import status +from rest_framework import viewsets +from rest_framework.response import Response + +from . import app_settings, paginators +from .models import Collection, CollectionItem, CollectionItemSnapshot, CollectionItemChunk +from .serializers import ( + CollectionSerializer, + CollectionItemSerializer, + CollectionItemSnapshotSerializer, + CollectionItemChunkSerializer + ) + + +User = get_user_model() + + +class BaseViewSet(viewsets.ModelViewSet): + authentication_classes = tuple(app_settings.API_AUTHENTICATORS) + permission_classes = tuple(app_settings.API_PERMISSIONS) + + def get_serializer_class(self): + serializer_class = self.serializer_class + + if self.request.method == 'PUT': + serializer_class = getattr(self, 'serializer_update_class', serializer_class) + + return serializer_class + + def get_collection_queryset(self, queryset=Collection.objects): + return queryset.all() + + +class CollectionViewSet(BaseViewSet): + allowed_methods = ['GET', 'POST', 'DELETE'] + permission_classes = BaseViewSet.permission_classes + queryset = Collection.objects.all() + serializer_class = CollectionSerializer + lookup_field = 'uid' + + def get_queryset(self): + queryset = type(self).queryset + return self.get_collection_queryset(queryset) + + def destroy(self, request, uid=None): + # FIXME: implement + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + def create(self, request, *args, **kwargs): + serializer = self.serializer_class(data=request.data) + if serializer.is_valid(): + try: + with transaction.atomic(): + serializer.save(owner=self.request.user) + except IntegrityError: + content = {'code': 'integrity_error'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + return Response({}, status=status.HTTP_201_CREATED) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + def list(self, request): + queryset = self.get_queryset() + + serializer = self.serializer_class(queryset, context={'request': request}, many=True) + return Response(serializer.data) + + +class CollectionItemViewSet(BaseViewSet): + allowed_methods = ['GET', 'POST'] + permission_classes = BaseViewSet.permission_classes + queryset = CollectionItem.objects.all() + serializer_class = CollectionItemSerializer + pagination_class = paginators.LinkHeaderPagination + lookup_field = 'uid' + + def get_queryset(self): + collection_uid = self.kwargs['collection_uid'] + try: + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + except Collection.DoesNotExist: + raise Http404("Collection does not exist") + # XXX Potentially add this for performance: .prefetch_related('snapshots__chunks') + queryset = type(self).queryset.filter(collection__pk=collection.pk) + + return queryset + + def create(self, request, collection_uid=None): + collection_object = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + + many = isinstance(request.data, list) + serializer = self.serializer_class(data=request.data, many=many) + if serializer.is_valid(): + try: + serializer.save(collection=collection_object) + except IntegrityError: + content = {'code': 'integrity_error'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + return Response({}, status=status.HTTP_201_CREATED) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + def destroy(self, request, collection_uid=None, uid=None): + # FIXME: implement + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + def update(self, request, collection_uid=None, uid=None): + # FIXME: implement, or should it be implemented elsewhere? + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + def partial_update(self, request, collection_uid=None, uid=None): + # FIXME: implement, or should it be implemented elsewhere? + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) From 0a3bb6f4bb47b38313b5e4b8ce7ca278eebf5b2f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 12:02:59 +0200 Subject: [PATCH 053/511] Merge item snapshot and item to be one model. --- .../migrations/0002_auto_20200220_0943.py | 53 +++++++++++++++++++ .../migrations/0003_collectionitem_current.py | 18 +++++++ .../migrations/0004_auto_20200220_1029.py | 18 +++++++ django_etesync/models.py | 32 ++++++----- django_etesync/serializers.py | 28 +++++----- django_etesync/views.py | 4 +- 6 files changed, 121 insertions(+), 32 deletions(-) create mode 100644 django_etesync/migrations/0002_auto_20200220_0943.py create mode 100644 django_etesync/migrations/0003_collectionitem_current.py create mode 100644 django_etesync/migrations/0004_auto_20200220_1029.py diff --git a/django_etesync/migrations/0002_auto_20200220_0943.py b/django_etesync/migrations/0002_auto_20200220_0943.py new file mode 100644 index 0000000..c150a11 --- /dev/null +++ b/django_etesync/migrations/0002_auto_20200220_0943.py @@ -0,0 +1,53 @@ +# Generated by Django 3.0.3 on 2020-02-20 09:43 + +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0001_initial'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitemchunk', + name='itemSnapshot', + ), + migrations.AddField( + model_name='collectionitem', + name='hmac', + field=models.CharField(default='', max_length=50), + preserve_default=False, + ), + migrations.AddField( + model_name='collectionitemchunk', + name='items', + field=models.ManyToManyField(related_name='chunks', to='django_etesync.CollectionItem'), + ), + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{44}')]), + ), + migrations.AlterField( + model_name='collectionitem', + name='collection', + field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='items', to='django_etesync.Collection'), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{44}')]), + ), + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{44}')]), + ), + migrations.DeleteModel( + name='CollectionItemSnapshot', + ), + ] diff --git a/django_etesync/migrations/0003_collectionitem_current.py b/django_etesync/migrations/0003_collectionitem_current.py new file mode 100644 index 0000000..2ffbf54 --- /dev/null +++ b/django_etesync/migrations/0003_collectionitem_current.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-20 09:47 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0002_auto_20200220_0943'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitem', + name='current', + field=models.BooleanField(default=True), + ), + ] diff --git a/django_etesync/migrations/0004_auto_20200220_1029.py b/django_etesync/migrations/0004_auto_20200220_1029.py new file mode 100644 index 0000000..1ea337a --- /dev/null +++ b/django_etesync/migrations/0004_auto_20200220_1029.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-20 10:29 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0003_collectionitem_current'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitem', + name='current', + field=models.BooleanField(db_index=True, default=True), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 9e578ba..f773495 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -12,6 +12,8 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . +from pathlib import Path + from django.db import models from django.conf import settings from django.core.validators import RegexValidator @@ -33,45 +35,41 @@ class Collection(models.Model): def __str__(self): return self.uid + @cached_property + def current_items(self): + return self.items.filter(current=True) + class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) version = models.PositiveSmallIntegerField() encryptionKey = models.BinaryField(editable=True, blank=False, null=False) - collection = models.ForeignKey(Collection, on_delete=models.CASCADE) + collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) + hmac = models.CharField(max_length=50, blank=False, null=False) + current = models.BooleanField(db_index=True, default=True) class Meta: unique_together = ('uid', 'collection') - @cached_property - def content(self): - return self.snapshots.get(current=True) - def __str__(self): return self.uid -class CollectionItemSnapshot(models.Model): - item = models.ForeignKey(CollectionItem, related_name='snapshots', on_delete=models.CASCADE) - current = models.BooleanField(default=True) - chunkHmac = models.CharField(max_length=50, blank=False, null=False) - - class Meta: - unique_together = ('item', 'current') - - def __str__(self): - return "{}, current={}".format(self.item.uid, self.current) +def chunk_directory_path(instance, filename): + col = instance.itemSnapshot.item.collection + user_id = col.owner.id + return Path('user_{}'.format(user_id), col.uid, instance.uid) class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) - itemSnapshot = models.ForeignKey(CollectionItemSnapshot, related_name='chunks', null=True, on_delete=models.SET_NULL) + items = models.ManyToManyField(CollectionItem, related_name='chunks') order = models.CharField(max_length=100, blank=False, null=False) + # We probably just want to implement this manually because we can have more than one pointing to a file. chunkFile = models.FileField(upload_to=chunk_directory_path) class Meta: - # unique_together = ('itemSnapshot', 'order') # Currently off because we set the item snapshot to null on deletion ordering = ['order'] def __str__(self): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index d4f259f..15034c2 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -64,25 +64,27 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): fields = ('uid', ) -class CollectionItemSnapshotSerializer(serializers.ModelSerializer): +class CollectionItemSerializer(serializers.ModelSerializer): + encryptionKey = BinaryBase64Field() chunks = serializers.SlugRelatedField( slug_field='uid', - queryset=models.CollectionItemChunk, + queryset=models.CollectionItemChunk.objects.all(), many=True ) class Meta: - model = models.CollectionItemSnapshot - fields = ('chunks', 'chunkHmac') + model = models.CollectionItem + fields = ('uid', 'version', 'encryptionKey', 'chunks', 'hmac') -class CollectionItemSerializer(serializers.ModelSerializer): - encryptionKey = BinaryBase64Field() - content = CollectionItemSnapshotSerializer( - read_only=True, - many=False - ) +class CollectionItemInlineSerializer(CollectionItemSerializer): + chunksData = serializers.SerializerMethodField('get_inline_chunks_from_context') - class Meta: - model = models.CollectionItem - fields = ('uid', 'version', 'encryptionKey', 'content') + class Meta(CollectionItemSerializer.Meta): + fields = CollectionItemSerializer.Meta.fields + ('chunksData', ) + + def get_inline_chunks_from_context(self, obj): + request = self.context.get('request', None) + if request is not None: + return ['SomeInlineData', 'Somemoredata'] + return 'readOnly' diff --git a/django_etesync/views.py b/django_etesync/views.py index 4d99dad..8724ae2 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -27,11 +27,11 @@ from rest_framework import viewsets from rest_framework.response import Response from . import app_settings, paginators -from .models import Collection, CollectionItem, CollectionItemSnapshot, CollectionItemChunk +from .models import Collection, CollectionItem, CollectionItemChunk from .serializers import ( CollectionSerializer, CollectionItemSerializer, - CollectionItemSnapshotSerializer, + CollectionItemInlineSerializer, CollectionItemChunkSerializer ) From 4075f775e76acbaf047d131e15c6eefe35b315de Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 12:30:20 +0200 Subject: [PATCH 054/511] Implement prefer-inline for fetching items. --- django_etesync/views.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 8724ae2..0c28974 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -99,6 +99,12 @@ class CollectionItemViewSet(BaseViewSet): pagination_class = paginators.LinkHeaderPagination lookup_field = 'uid' + def get_serializer_class(self): + if self.request.method == 'GET' and self.request.query_params.get('prefer_inline'): + return CollectionItemInlineSerializer + + return super().get_serializer_class() + def get_queryset(self): collection_uid = self.kwargs['collection_uid'] try: From 67fb714ddba00af185f43cdadc47069c7c559958 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 13:56:16 +0200 Subject: [PATCH 055/511] More progress. --- .../migrations/0005_auto_20200220_1123.py | 29 +++++++++++ .../migrations/0006_auto_20200220_1137.py | 49 +++++++++++++++++++ .../migrations/0007_auto_20200220_1144.py | 28 +++++++++++ django_etesync/models.py | 40 ++++++++++----- django_etesync/serializers.py | 24 ++++++--- django_etesync/views.py | 38 ++++++++++++++ 6 files changed, 189 insertions(+), 19 deletions(-) create mode 100644 django_etesync/migrations/0005_auto_20200220_1123.py create mode 100644 django_etesync/migrations/0006_auto_20200220_1137.py create mode 100644 django_etesync/migrations/0007_auto_20200220_1144.py diff --git a/django_etesync/migrations/0005_auto_20200220_1123.py b/django_etesync/migrations/0005_auto_20200220_1123.py new file mode 100644 index 0000000..88c9ea6 --- /dev/null +++ b/django_etesync/migrations/0005_auto_20200220_1123.py @@ -0,0 +1,29 @@ +# Generated by Django 3.0.3 on 2020-02-20 11:23 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0004_auto_20200220_1029'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitemchunk', + name='items', + ), + migrations.AddField( + model_name='collectionitem', + name='chunks', + field=models.ManyToManyField(related_name='items', to='django_etesync.CollectionItemChunk'), + ), + migrations.AddField( + model_name='collectionitemchunk', + name='collection', + field=models.ForeignKey(default=1, on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etesync.Collection'), + preserve_default=False, + ), + ] diff --git a/django_etesync/migrations/0006_auto_20200220_1137.py b/django_etesync/migrations/0006_auto_20200220_1137.py new file mode 100644 index 0000000..efc421e --- /dev/null +++ b/django_etesync/migrations/0006_auto_20200220_1137.py @@ -0,0 +1,49 @@ +# Generated by Django 3.0.3 on 2020-02-20 11:37 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0005_auto_20200220_1123'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitem', + name='chunks', + ), + migrations.RemoveField( + model_name='collectionitem', + name='current', + ), + migrations.RemoveField( + model_name='collectionitem', + name='encryptionKey', + ), + migrations.RemoveField( + model_name='collectionitem', + name='hmac', + ), + migrations.RemoveField( + model_name='collectionitem', + name='version', + ), + migrations.CreateModel( + name='CollectionItemSnapshot', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('version', models.PositiveSmallIntegerField()), + ('encryptionKey', models.BinaryField(editable=True)), + ('hmac', models.CharField(max_length=50)), + ('current', models.BooleanField(db_index=True, default=True)), + ('chunks', models.ManyToManyField(related_name='items', to='django_etesync.CollectionItemChunk')), + ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='snapshots', to='django_etesync.CollectionItem')), + ], + options={ + 'unique_together': {('item', 'current')}, + }, + ), + ] diff --git a/django_etesync/migrations/0007_auto_20200220_1144.py b/django_etesync/migrations/0007_auto_20200220_1144.py new file mode 100644 index 0000000..3ebf55b --- /dev/null +++ b/django_etesync/migrations/0007_auto_20200220_1144.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.3 on 2020-02-20 11:44 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0006_auto_20200220_1137'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemchunk', + name='item', + field=models.ForeignKey(default=1, on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etesync.CollectionItem'), + preserve_default=False, + ), + migrations.AlterUniqueTogether( + name='collectionitemchunk', + unique_together={('item', 'order')}, + ), + migrations.RemoveField( + model_name='collectionitemchunk', + name='collection', + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index f773495..1bd2090 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -35,19 +35,18 @@ class Collection(models.Model): def __str__(self): return self.uid - @cached_property - def current_items(self): - return self.items.filter(current=True) + +def chunk_directory_path(instance, filename): + col = instance.itemSnapshot.item.collection + user_id = col.owner.id + return Path('user_{}'.format(user_id), col.uid, instance.uid) + class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) - version = models.PositiveSmallIntegerField() - encryptionKey = models.BinaryField(editable=True, blank=False, null=False) collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) - hmac = models.CharField(max_length=50, blank=False, null=False) - current = models.BooleanField(db_index=True, default=True) class Meta: unique_together = ('uid', 'collection') @@ -55,22 +54,37 @@ class CollectionItem(models.Model): def __str__(self): return self.uid - -def chunk_directory_path(instance, filename): - col = instance.itemSnapshot.item.collection - user_id = col.owner.id - return Path('user_{}'.format(user_id), col.uid, instance.uid) + @cached_property + def content(self): + return self.snapshots.get(current=True) class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) - items = models.ManyToManyField(CollectionItem, related_name='chunks') + item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) order = models.CharField(max_length=100, blank=False, null=False) # We probably just want to implement this manually because we can have more than one pointing to a file. chunkFile = models.FileField(upload_to=chunk_directory_path) class Meta: + unique_together = ('item', 'order') ordering = ['order'] def __str__(self): return self.uid + + +class CollectionItemSnapshot(models.Model): + version = models.PositiveSmallIntegerField() + encryptionKey = models.BinaryField(editable=True, blank=False, null=False) + item = models.ForeignKey(CollectionItem, related_name='snapshots', on_delete=models.CASCADE) + chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') + hmac = models.CharField(max_length=50, blank=False, null=False) + current = models.BooleanField(db_index=True, default=True) + + class Meta: + unique_together = ('item', 'current') + + def __str__(self): + return '{} {} current={}'.format(self.item.uid, self.id, self.current) + diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 15034c2..f86d5e1 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -64,7 +64,7 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): fields = ('uid', ) -class CollectionItemSerializer(serializers.ModelSerializer): +class CollectionItemSnapshotSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() chunks = serializers.SlugRelatedField( slug_field='uid', @@ -73,18 +73,30 @@ class CollectionItemSerializer(serializers.ModelSerializer): ) class Meta: - model = models.CollectionItem - fields = ('uid', 'version', 'encryptionKey', 'chunks', 'hmac') + model = models.CollectionItemSnapshot + fields = ('version', 'encryptionKey', 'chunks', 'hmac') -class CollectionItemInlineSerializer(CollectionItemSerializer): +class CollectionItemSnapshotInlineSerializer(CollectionItemSnapshotSerializer): chunksData = serializers.SerializerMethodField('get_inline_chunks_from_context') - class Meta(CollectionItemSerializer.Meta): - fields = CollectionItemSerializer.Meta.fields + ('chunksData', ) + class Meta(CollectionItemSnapshotSerializer.Meta): + fields = CollectionItemSnapshotSerializer.Meta.fields + ('chunksData', ) def get_inline_chunks_from_context(self, obj): request = self.context.get('request', None) if request is not None: return ['SomeInlineData', 'Somemoredata'] return 'readOnly' + + +class CollectionItemSerializer(serializers.ModelSerializer): + content = CollectionItemSnapshotSerializer(read_only=True, many=False) + + class Meta: + model = models.CollectionItem + fields = ('uid', 'content') + + +class CollectionItemInlineSerializer(CollectionItemSerializer): + content = CollectionItemSnapshotInlineSerializer(read_only=True, many=False) diff --git a/django_etesync/views.py b/django_etesync/views.py index 0c28974..57136a4 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -24,6 +24,7 @@ from django.views.decorators.http import require_POST from rest_framework import status from rest_framework import viewsets +from rest_framework import parsers from rest_framework.response import Response from . import app_settings, paginators @@ -143,3 +144,40 @@ class CollectionItemViewSet(BaseViewSet): def partial_update(self, request, collection_uid=None, uid=None): # FIXME: implement, or should it be implemented elsewhere? return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + +class CollectionItemChunkViewSet(viewsets.ViewSet): + allowed_methods = ['GET', 'POST'] + authentication_classes = BaseViewSet.authentication_classes + permission_classes = BaseViewSet.permission_classes + parser_classes = (parsers.MultiPartParser, ) + lookup_field = 'uid' + + def create(self, request, collection_uid=None): + # FIXME: we are potentially not getting the correct queryset + collection_object = Collection.objects.get(uid=collection_uid) + + many = isinstance(request.data, list) + serializer = self.serializer_class(data=request.data, many=many) + if serializer.is_valid(): + try: + serializer.save(collection=collection_object) + except IntegrityError: + content = {'code': 'integrity_error'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + return Response({}, status=status.HTTP_201_CREATED) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + def destroy(self, request, collection_uid=None, uid=None): + # FIXME: implement + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + def update(self, request, collection_uid=None, uid=None): + # FIXME: implement, or should it be implemented elsewhere? + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + def partial_update(self, request, collection_uid=None, uid=None): + # FIXME: implement, or should it be implemented elsewhere? + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) From 0c44f738fdff2f16064d782cd15a2a3e3b1021a7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 14:42:35 +0200 Subject: [PATCH 056/511] More progress - support chunk uploading. --- .../0008_collectionitemchunk_chunkfile.py | 20 ++++++++++ .../migrations/0009_auto_20200220_1220.py | 19 +++++++++ django_etesync/models.py | 16 ++++---- django_etesync/serializers.py | 40 ++++++++++++++----- django_etesync/views.py | 25 ++++-------- 5 files changed, 83 insertions(+), 37 deletions(-) create mode 100644 django_etesync/migrations/0008_collectionitemchunk_chunkfile.py create mode 100644 django_etesync/migrations/0009_auto_20200220_1220.py diff --git a/django_etesync/migrations/0008_collectionitemchunk_chunkfile.py b/django_etesync/migrations/0008_collectionitemchunk_chunkfile.py new file mode 100644 index 0000000..68bb27c --- /dev/null +++ b/django_etesync/migrations/0008_collectionitemchunk_chunkfile.py @@ -0,0 +1,20 @@ +# Generated by Django 3.0.3 on 2020-02-20 12:16 + +from django.db import migrations, models +import django_etesync.models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0007_auto_20200220_1144'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemchunk', + name='chunkFile', + field=models.FileField(default='', upload_to=django_etesync.models.chunk_directory_path), + preserve_default=False, + ), + ] diff --git a/django_etesync/migrations/0009_auto_20200220_1220.py b/django_etesync/migrations/0009_auto_20200220_1220.py new file mode 100644 index 0000000..71e1539 --- /dev/null +++ b/django_etesync/migrations/0009_auto_20200220_1220.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-02-20 12:20 + +from django.db import migrations, models +import django_etesync.models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0008_collectionitemchunk_chunkfile'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemchunk', + name='chunkFile', + field=models.FileField(max_length=150, upload_to=django_etesync.models.chunk_directory_path), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 1bd2090..0cefbee 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -36,12 +36,6 @@ class Collection(models.Model): return self.uid -def chunk_directory_path(instance, filename): - col = instance.itemSnapshot.item.collection - user_id = col.owner.id - return Path('user_{}'.format(user_id), col.uid, instance.uid) - - class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, @@ -59,12 +53,19 @@ class CollectionItem(models.Model): return self.snapshots.get(current=True) +def chunk_directory_path(instance, filename): + item = instance.item + col = item.collection + user_id = col.owner.id + return Path('user_{}'.format(user_id), col.uid, item.uid, instance.uid) + + class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) order = models.CharField(max_length=100, blank=False, null=False) - # We probably just want to implement this manually because we can have more than one pointing to a file. chunkFile = models.FileField(upload_to=chunk_directory_path) + chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150) class Meta: unique_together = ('item', 'order') @@ -87,4 +88,3 @@ class CollectionItemSnapshot(models.Model): def __str__(self): return '{} {} current={}'.format(self.item.uid, self.id, self.current) - diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index f86d5e1..a4d6fbe 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -61,10 +61,10 @@ class CollectionSerializer(serializers.ModelSerializer): class CollectionItemChunkSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemChunk - fields = ('uid', ) + fields = ('uid', 'chunkFile') -class CollectionItemSnapshotSerializer(serializers.ModelSerializer): +class CollectionItemSnapshotBaseSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() chunks = serializers.SlugRelatedField( slug_field='uid', @@ -77,18 +77,36 @@ class CollectionItemSnapshotSerializer(serializers.ModelSerializer): fields = ('version', 'encryptionKey', 'chunks', 'hmac') -class CollectionItemSnapshotInlineSerializer(CollectionItemSnapshotSerializer): - chunksData = serializers.SerializerMethodField('get_inline_chunks_from_context') +class CollectionItemSnapshotSerializer(CollectionItemSnapshotBaseSerializer): + chunksUrls = serializers.SerializerMethodField('get_chunks_urls') - class Meta(CollectionItemSnapshotSerializer.Meta): - fields = CollectionItemSnapshotSerializer.Meta.fields + ('chunksData', ) + class Meta(CollectionItemSnapshotBaseSerializer.Meta): + fields = CollectionItemSnapshotBaseSerializer.Meta.fields + ('chunksUrls', ) - def get_inline_chunks_from_context(self, obj): - request = self.context.get('request', None) - if request is not None: - return ['SomeInlineData', 'Somemoredata'] - return 'readOnly' + # FIXME: currently the user is exposed in the url. We don't want that, and we can probably avoid that but still save it under the user. + # We would probably be better off just let the user calculate the urls from the uid and a base url for the snapshot. + # E.g. chunkBaseUrl: "/media/bla/bla/" or chunkBaseUrl: "https://media.etesync.com/bla/bla" + def get_chunks_urls(self, obj): + ret = [] + for chunk in obj.chunks.all(): + ret.append(chunk.chunkFile.url) + + return ret + + +class CollectionItemSnapshotInlineSerializer(CollectionItemSnapshotBaseSerializer): + chunksData = serializers.SerializerMethodField('get_chunks_data') + + class Meta(CollectionItemSnapshotBaseSerializer.Meta): + fields = CollectionItemSnapshotBaseSerializer.Meta.fields + ('chunksData', ) + + def get_chunks_data(self, obj): + ret = [] + for chunk in obj.chunks.all(): + with open(chunk.chunkFile.path, 'rb') as f: + ret.append(base64.b64encode(f.read()).decode('ascii')) + return ret class CollectionItemSerializer(serializers.ModelSerializer): content = CollectionItemSnapshotSerializer(read_only=True, many=False) diff --git a/django_etesync/views.py b/django_etesync/views.py index 57136a4..5caa452 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -148,20 +148,21 @@ class CollectionItemViewSet(BaseViewSet): class CollectionItemChunkViewSet(viewsets.ViewSet): allowed_methods = ['GET', 'POST'] + parser_classes = (parsers.MultiPartParser, ) authentication_classes = BaseViewSet.authentication_classes permission_classes = BaseViewSet.permission_classes - parser_classes = (parsers.MultiPartParser, ) + serializer_class = CollectionItemChunkSerializer lookup_field = 'uid' - def create(self, request, collection_uid=None): + def create(self, request, collection_uid=None, collection_item_uid=None): # FIXME: we are potentially not getting the correct queryset - collection_object = Collection.objects.get(uid=collection_uid) + col = get_object_or_404(Collection.objects, uid=collection_uid) + col_it = get_object_or_404(col.items, uid=collection_item_uid) - many = isinstance(request.data, list) - serializer = self.serializer_class(data=request.data, many=many) + serializer = self.serializer_class(data=request.data) if serializer.is_valid(): try: - serializer.save(collection=collection_object) + serializer.save(item=col_it, order='abc') except IntegrityError: content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) @@ -169,15 +170,3 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_201_CREATED) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) - - def destroy(self, request, collection_uid=None, uid=None): - # FIXME: implement - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def update(self, request, collection_uid=None, uid=None): - # FIXME: implement, or should it be implemented elsewhere? - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def partial_update(self, request, collection_uid=None, uid=None): - # FIXME: implement, or should it be implemented elsewhere? - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) From d57ed0341712f47b69abf9dc870a68c40f900f2e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 14:48:19 +0200 Subject: [PATCH 057/511] Make sure we don't upload the same file twice. --- .../migrations/0010_auto_20200220_1248.py | 19 +++++++++++++++++++ django_etesync/models.py | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0010_auto_20200220_1248.py diff --git a/django_etesync/migrations/0010_auto_20200220_1248.py b/django_etesync/migrations/0010_auto_20200220_1248.py new file mode 100644 index 0000000..0c08ed0 --- /dev/null +++ b/django_etesync/migrations/0010_auto_20200220_1248.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-02-20 12:48 + +from django.db import migrations, models +import django_etesync.models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0009_auto_20200220_1220'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemchunk', + name='chunkFile', + field=models.FileField(max_length=150, unique=True, upload_to=django_etesync.models.chunk_directory_path), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 0cefbee..37c0dc1 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -65,7 +65,7 @@ class CollectionItemChunk(models.Model): max_length=44, validators=[UidValidator]) item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) order = models.CharField(max_length=100, blank=False, null=False) - chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150) + chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) class Meta: unique_together = ('item', 'order') From b17e944dd27e733463f8ed92ecadedf573eca79d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 15:39:52 +0200 Subject: [PATCH 058/511] Make it possible to download the chunk from the rest API. --- django_etesync/views.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 5caa452..9d816ec 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -25,6 +25,7 @@ from django.views.decorators.http import require_POST from rest_framework import status from rest_framework import viewsets from rest_framework import parsers +from rest_framework.decorators import action as action_decorator from rest_framework.response import Response from . import app_settings, paginators @@ -170,3 +171,19 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_201_CREATED) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + @action_decorator(detail=True, methods=['GET']) + def download(self, request, collection_uid=None, collection_item_uid=None, uid=None): + import os + from django.views.static import serve + + col = get_object_or_404(Collection.objects, uid=collection_uid) + col_it = get_object_or_404(col.items, uid=collection_item_uid) + chunk = get_object_or_404(col_it.chunks, uid=uid) + + filename = chunk.chunkFile.path + dirname = os.path.dirname(filename) + basename = os.path.basename(filename) + + # FIXME: DO NOT USE! Use django-send file or etc instead. + return serve(request, basename, dirname) From 24cb6ed6ee9d7931503f18352819313931739274 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 16:35:20 +0200 Subject: [PATCH 059/511] Also serve an item's snapshots. --- django_etesync/views.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 9d816ec..f44f79a 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -34,6 +34,8 @@ from .serializers import ( CollectionSerializer, CollectionItemSerializer, CollectionItemInlineSerializer, + CollectionItemSnapshotSerializer, + CollectionItemSnapshotInlineSerializer, CollectionItemChunkSerializer ) @@ -146,6 +148,14 @@ class CollectionItemViewSet(BaseViewSet): # FIXME: implement, or should it be implemented elsewhere? return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + @action_decorator(detail=True, methods=['GET']) + def snapshots(self, request, collection_uid=None, uid=None): + col = get_object_or_404(Collection.objects, uid=collection_uid) + col_it = get_object_or_404(col.items, uid=uid) + + serializer = CollectionItemSnapshotSerializer(col_it.snapshots, many=True) + return Response(serializer.data) + class CollectionItemChunkViewSet(viewsets.ViewSet): allowed_methods = ['GET', 'POST'] From 0a40a04d3bc0ec765f6be36b46bcfd55056207fb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 17:33:34 +0200 Subject: [PATCH 060/511] Chunk view: unify how we get the wanted collection queryset. --- django_etesync/views.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index f44f79a..1e97b4f 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -165,9 +165,11 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer_class = CollectionItemChunkSerializer lookup_field = 'uid' + def get_collection_queryset(self, queryset=Collection.objects): + return queryset.all() + def create(self, request, collection_uid=None, collection_item_uid=None): - # FIXME: we are potentially not getting the correct queryset - col = get_object_or_404(Collection.objects, uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) serializer = self.serializer_class(data=request.data) @@ -187,7 +189,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): import os from django.views.static import serve - col = get_object_or_404(Collection.objects, uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) chunk = get_object_or_404(col_it.chunks, uid=uid) From c3fc00b9d8a7092fd2a329bfd6433381285de4f4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 17:34:51 +0200 Subject: [PATCH 061/511] Add a FIXME. --- django_etesync/views.py | 1 + 1 file changed, 1 insertion(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 1e97b4f..e7a6f87 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -175,6 +175,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer = self.serializer_class(data=request.data) if serializer.is_valid(): try: + # FIXME: actually generate the correct order value. Or alternatively have it null at first and only set it when ommitting to a snapshot serializer.save(item=col_it, order='abc') except IntegrityError: content = {'code': 'integrity_error'} From 052483d38c7776ac0b74c9cde73ecd5b82001222 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 18:41:07 +0200 Subject: [PATCH 062/511] Serve snapshots newest to oldest. --- django_etesync/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index e7a6f87..8e21e3c 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -153,7 +153,7 @@ class CollectionItemViewSet(BaseViewSet): col = get_object_or_404(Collection.objects, uid=collection_uid) col_it = get_object_or_404(col.items, uid=uid) - serializer = CollectionItemSnapshotSerializer(col_it.snapshots, many=True) + serializer = CollectionItemSnapshotSerializer(col_it.snapshots.order_by('-id'), many=True) return Response(serializer.data) From cc00391504889799fae3ca9670e53978e445443b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 20 Feb 2020 22:41:39 +0200 Subject: [PATCH 063/511] Rename Snapshot to Revision --- .../migrations/0011_auto_20200220_2037.py | 17 ++++++++++++++++ .../migrations/0012_auto_20200220_2038.py | 19 ++++++++++++++++++ django_etesync/models.py | 6 +++--- django_etesync/serializers.py | 20 +++++++++---------- django_etesync/views.py | 10 +++++----- 5 files changed, 54 insertions(+), 18 deletions(-) create mode 100644 django_etesync/migrations/0011_auto_20200220_2037.py create mode 100644 django_etesync/migrations/0012_auto_20200220_2038.py diff --git a/django_etesync/migrations/0011_auto_20200220_2037.py b/django_etesync/migrations/0011_auto_20200220_2037.py new file mode 100644 index 0000000..2d790748 --- /dev/null +++ b/django_etesync/migrations/0011_auto_20200220_2037.py @@ -0,0 +1,17 @@ +# Generated by Django 3.0.3 on 2020-02-20 20:37 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0010_auto_20200220_1248'), + ] + + operations = [ + migrations.RenameModel( + old_name='CollectionItemSnapshot', + new_name='CollectionItemRevision', + ), + ] diff --git a/django_etesync/migrations/0012_auto_20200220_2038.py b/django_etesync/migrations/0012_auto_20200220_2038.py new file mode 100644 index 0000000..2657973 --- /dev/null +++ b/django_etesync/migrations/0012_auto_20200220_2038.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-02-20 20:38 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0011_auto_20200220_2037'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemrevision', + name='item', + field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='django_etesync.CollectionItem'), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 37c0dc1..0de3f90 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -50,7 +50,7 @@ class CollectionItem(models.Model): @cached_property def content(self): - return self.snapshots.get(current=True) + return self.revisions.get(current=True) def chunk_directory_path(instance, filename): @@ -75,10 +75,10 @@ class CollectionItemChunk(models.Model): return self.uid -class CollectionItemSnapshot(models.Model): +class CollectionItemRevision(models.Model): version = models.PositiveSmallIntegerField() encryptionKey = models.BinaryField(editable=True, blank=False, null=False) - item = models.ForeignKey(CollectionItem, related_name='snapshots', on_delete=models.CASCADE) + item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) current = models.BooleanField(db_index=True, default=True) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index a4d6fbe..de49c98 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -64,7 +64,7 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): fields = ('uid', 'chunkFile') -class CollectionItemSnapshotBaseSerializer(serializers.ModelSerializer): +class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() chunks = serializers.SlugRelatedField( slug_field='uid', @@ -73,15 +73,15 @@ class CollectionItemSnapshotBaseSerializer(serializers.ModelSerializer): ) class Meta: - model = models.CollectionItemSnapshot + model = models.CollectionItemRevision fields = ('version', 'encryptionKey', 'chunks', 'hmac') -class CollectionItemSnapshotSerializer(CollectionItemSnapshotBaseSerializer): +class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): chunksUrls = serializers.SerializerMethodField('get_chunks_urls') - class Meta(CollectionItemSnapshotBaseSerializer.Meta): - fields = CollectionItemSnapshotBaseSerializer.Meta.fields + ('chunksUrls', ) + class Meta(CollectionItemRevisionBaseSerializer.Meta): + fields = CollectionItemRevisionBaseSerializer.Meta.fields + ('chunksUrls', ) # FIXME: currently the user is exposed in the url. We don't want that, and we can probably avoid that but still save it under the user. # We would probably be better off just let the user calculate the urls from the uid and a base url for the snapshot. @@ -94,11 +94,11 @@ class CollectionItemSnapshotSerializer(CollectionItemSnapshotBaseSerializer): return ret -class CollectionItemSnapshotInlineSerializer(CollectionItemSnapshotBaseSerializer): +class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerializer): chunksData = serializers.SerializerMethodField('get_chunks_data') - class Meta(CollectionItemSnapshotBaseSerializer.Meta): - fields = CollectionItemSnapshotBaseSerializer.Meta.fields + ('chunksData', ) + class Meta(CollectionItemRevisionBaseSerializer.Meta): + fields = CollectionItemRevisionBaseSerializer.Meta.fields + ('chunksData', ) def get_chunks_data(self, obj): ret = [] @@ -109,7 +109,7 @@ class CollectionItemSnapshotInlineSerializer(CollectionItemSnapshotBaseSerialize return ret class CollectionItemSerializer(serializers.ModelSerializer): - content = CollectionItemSnapshotSerializer(read_only=True, many=False) + content = CollectionItemRevisionSerializer(read_only=True, many=False) class Meta: model = models.CollectionItem @@ -117,4 +117,4 @@ class CollectionItemSerializer(serializers.ModelSerializer): class CollectionItemInlineSerializer(CollectionItemSerializer): - content = CollectionItemSnapshotInlineSerializer(read_only=True, many=False) + content = CollectionItemRevisionInlineSerializer(read_only=True, many=False) diff --git a/django_etesync/views.py b/django_etesync/views.py index 8e21e3c..d8604cc 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -34,8 +34,8 @@ from .serializers import ( CollectionSerializer, CollectionItemSerializer, CollectionItemInlineSerializer, - CollectionItemSnapshotSerializer, - CollectionItemSnapshotInlineSerializer, + CollectionItemRevisionSerializer, + CollectionItemRevisionInlineSerializer, CollectionItemChunkSerializer ) @@ -115,7 +115,7 @@ class CollectionItemViewSet(BaseViewSet): collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) except Collection.DoesNotExist: raise Http404("Collection does not exist") - # XXX Potentially add this for performance: .prefetch_related('snapshots__chunks') + # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = type(self).queryset.filter(collection__pk=collection.pk) return queryset @@ -149,11 +149,11 @@ class CollectionItemViewSet(BaseViewSet): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) @action_decorator(detail=True, methods=['GET']) - def snapshots(self, request, collection_uid=None, uid=None): + def revision(self, request, collection_uid=None, uid=None): col = get_object_or_404(Collection.objects, uid=collection_uid) col_it = get_object_or_404(col.items, uid=uid) - serializer = CollectionItemSnapshotSerializer(col_it.snapshots.order_by('-id'), many=True) + serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) return Response(serializer.data) From d6df94facf6c5015ef2373c2739c9bd3c4fe09c2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 14:20:23 +0200 Subject: [PATCH 064/511] Item create: 404 if collection isn't found. It doesn't actually change anything beacuse it 404s in the collection getting, but still, good to have this here too. --- django_etesync/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index d8604cc..4c744bd 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -121,7 +121,7 @@ class CollectionItemViewSet(BaseViewSet): return queryset def create(self, request, collection_uid=None): - collection_object = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) many = isinstance(request.data, list) serializer = self.serializer_class(data=request.data, many=many) From 358c59f6d75d90892732d8ff5860feaa6dbf3bb5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 14:20:52 +0200 Subject: [PATCH 065/511] Item: add bulk_get and a note about bulk creating. --- django_etesync/views.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 4c744bd..9150db0 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -123,6 +123,8 @@ class CollectionItemViewSet(BaseViewSet): def create(self, request, collection_uid=None): collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + # FIXME: change this to also support bulk update, or have another endpoint for that. + # See https://www.django-rest-framework.org/api-guide/serializers/#customizing-multiple-update many = isinstance(request.data, list) serializer = self.serializer_class(data=request.data, many=many) if serializer.is_valid(): @@ -156,6 +158,16 @@ class CollectionItemViewSet(BaseViewSet): serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) return Response(serializer.data) + @action_decorator(detail=False, methods=['POST']) + def bulk_get(self, request, collection_uid=None): + queryset = self.get_queryset() + + if isinstance(request.data, list): + queryset = queryset.filter(uid__in=request.data) + + serializer = self.get_serializer_class()(queryset, many=True) + return Response(serializer.data, status=status.HTTP_200_OK) + class CollectionItemChunkViewSet(viewsets.ViewSet): allowed_methods = ['GET', 'POST'] From 0beaaf5bf95fc6b7411d92a10d168b9d68f798db Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 14:21:14 +0200 Subject: [PATCH 066/511] lint: fix many pylint warnings. --- django_etesync/serializers.py | 4 +++- django_etesync/views.py | 15 +++++---------- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index de49c98..0e5b355 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -83,7 +83,8 @@ class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): class Meta(CollectionItemRevisionBaseSerializer.Meta): fields = CollectionItemRevisionBaseSerializer.Meta.fields + ('chunksUrls', ) - # FIXME: currently the user is exposed in the url. We don't want that, and we can probably avoid that but still save it under the user. + # FIXME: currently the user is exposed in the url. We don't want that, and we can probably avoid that but still + # save it under the user. # We would probably be better off just let the user calculate the urls from the uid and a base url for the snapshot. # E.g. chunkBaseUrl: "/media/bla/bla/" or chunkBaseUrl: "https://media.etesync.com/bla/bla" def get_chunks_urls(self, obj): @@ -108,6 +109,7 @@ class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerialize return ret + class CollectionItemSerializer(serializers.ModelSerializer): content = CollectionItemRevisionSerializer(read_only=True, many=False) diff --git a/django_etesync/views.py b/django_etesync/views.py index 9150db0..2e91261 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -12,15 +12,10 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -from django.conf import settings -from django.contrib.auth import login, get_user_model +from django.contrib.auth import get_user_model from django.db import IntegrityError, transaction -from django.db.models import Q -from django.core.exceptions import ObjectDoesNotExist -from django.http import HttpResponseBadRequest, HttpResponse, Http404 +from django.http import Http404 from django.shortcuts import get_object_or_404 -from django.views.decorators.csrf import csrf_exempt -from django.views.decorators.http import require_POST from rest_framework import status from rest_framework import viewsets @@ -29,13 +24,12 @@ from rest_framework.decorators import action as action_decorator from rest_framework.response import Response from . import app_settings, paginators -from .models import Collection, CollectionItem, CollectionItemChunk +from .models import Collection, CollectionItem from .serializers import ( CollectionSerializer, CollectionItemSerializer, CollectionItemInlineSerializer, CollectionItemRevisionSerializer, - CollectionItemRevisionInlineSerializer, CollectionItemChunkSerializer ) @@ -187,7 +181,8 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer = self.serializer_class(data=request.data) if serializer.is_valid(): try: - # FIXME: actually generate the correct order value. Or alternatively have it null at first and only set it when ommitting to a snapshot + # FIXME: actually generate the correct order value. Or alternatively have it null at first and only + # set it when ommitting to a snapshot serializer.save(item=col_it, order='abc') except IntegrityError: content = {'code': 'integrity_error'} From 727cd3e5faf45af298de747823820d7fda619158 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 14:32:25 +0200 Subject: [PATCH 067/511] pylint: fix more warnings. --- django_etesync/app_settings.py | 12 +++++------- django_etesync/models.py | 1 - 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/django_etesync/app_settings.py b/django_etesync/app_settings.py index 6a04b4e..a0a0d99 100644 --- a/django_etesync/app_settings.py +++ b/django_etesync/app_settings.py @@ -12,8 +12,6 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -import sys # noqa - class AppSettings: def __init__(self, prefix): @@ -22,17 +20,17 @@ class AppSettings: def import_from_str(self, name): from importlib import import_module - p, m = name.rsplit('.', 1) + path, prop = name.rsplit('.', 1) - mod = import_module(p) - return getattr(mod, m) + mod = import_module(path) + return getattr(mod, prop) def _setting(self, name, dflt): from django.conf import settings return getattr(settings, self.prefix + name, dflt) @property - def API_PERMISSIONS(self): + def API_PERMISSIONS(self): # pylint: disable=invalid-name perms = self._setting("API_PERMISSIONS", ('rest_framework.permissions.IsAuthenticated', )) ret = [] for perm in perms: @@ -40,7 +38,7 @@ class AppSettings: return ret @property - def API_AUTHENTICATORS(self): + def API_AUTHENTICATORS(self): # pylint: disable=invalid-name perms = self._setting("API_AUTHENTICATORS", ('rest_framework.authentication.TokenAuthentication', 'rest_framework.authentication.SessionAuthentication')) ret = [] diff --git a/django_etesync/models.py b/django_etesync/models.py index 0de3f90..18743fd 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -36,7 +36,6 @@ class Collection(models.Model): return self.uid - class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) From 4054a2f78cad318b2d45022e07529c7f7b2dac21 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 15:53:25 +0200 Subject: [PATCH 068/511] Implement item update and deletion. Deletion is essentially an update with "isDeletion" set to True. --- ...0013_collectionitemrevision_is_deletion.py | 18 +++++++++++++++ .../migrations/0014_auto_20200226_1322.py | 18 +++++++++++++++ .../migrations/0015_auto_20200226_1349.py | 18 +++++++++++++++ django_etesync/models.py | 3 ++- django_etesync/serializers.py | 22 +++++++++++++++++-- django_etesync/views.py | 8 ++----- 6 files changed, 78 insertions(+), 9 deletions(-) create mode 100644 django_etesync/migrations/0013_collectionitemrevision_is_deletion.py create mode 100644 django_etesync/migrations/0014_auto_20200226_1322.py create mode 100644 django_etesync/migrations/0015_auto_20200226_1349.py diff --git a/django_etesync/migrations/0013_collectionitemrevision_is_deletion.py b/django_etesync/migrations/0013_collectionitemrevision_is_deletion.py new file mode 100644 index 0000000..27f4953 --- /dev/null +++ b/django_etesync/migrations/0013_collectionitemrevision_is_deletion.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-26 13:17 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0012_auto_20200220_2038'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemrevision', + name='is_deletion', + field=models.BooleanField(default=False), + ), + ] diff --git a/django_etesync/migrations/0014_auto_20200226_1322.py b/django_etesync/migrations/0014_auto_20200226_1322.py new file mode 100644 index 0000000..1937015 --- /dev/null +++ b/django_etesync/migrations/0014_auto_20200226_1322.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-26 13:22 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0013_collectionitemrevision_is_deletion'), + ] + + operations = [ + migrations.RenameField( + model_name='collectionitemrevision', + old_name='is_deletion', + new_name='isDeletion', + ), + ] diff --git a/django_etesync/migrations/0015_auto_20200226_1349.py b/django_etesync/migrations/0015_auto_20200226_1349.py new file mode 100644 index 0000000..896619d --- /dev/null +++ b/django_etesync/migrations/0015_auto_20200226_1349.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-26 13:49 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0014_auto_20200226_1322'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemrevision', + name='current', + field=models.BooleanField(blank=True, db_index=True, default=True, null=True), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 18743fd..dda081f 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -80,7 +80,8 @@ class CollectionItemRevision(models.Model): item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) - current = models.BooleanField(db_index=True, default=True) + current = models.BooleanField(db_index=True, default=True, blank=True, null=True) + isDeletion = models.BooleanField(default=False) class Meta: unique_together = ('item', 'current') diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 0e5b355..b3f7254 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -15,6 +15,7 @@ import base64 from django.contrib.auth import get_user_model +from django.db import transaction from rest_framework import serializers from . import models @@ -74,7 +75,7 @@ class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemRevision - fields = ('version', 'encryptionKey', 'chunks', 'hmac') + fields = ('version', 'encryptionKey', 'chunks', 'hmac', 'isDeletion') class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): @@ -111,12 +112,29 @@ class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerialize class CollectionItemSerializer(serializers.ModelSerializer): - content = CollectionItemRevisionSerializer(read_only=True, many=False) + content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.CollectionItem fields = ('uid', 'content') + def update(self, instance, validated_data): + """Function that's called when this serializer is meant to update an item""" + revision_data = validated_data.pop('content') + + with transaction.atomic(): + # We don't have to use select_for_update here because the unique constraint on current guards against + # the race condition. But it's a good idea because it'll lock and wait rather than fail. + current_revision = instance.revisions.filter(current=True).select_for_update().first() + current_revision.current = None + current_revision.save() + + chunks = revision_data.pop('chunks') + revision = models.CollectionItemRevision.objects.create(**revision_data, item=instance) + revision.chunks.set(chunks) + + return instance + class CollectionItemInlineSerializer(CollectionItemSerializer): content = CollectionItemRevisionInlineSerializer(read_only=True, many=False) diff --git a/django_etesync/views.py b/django_etesync/views.py index 2e91261..b90b62e 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -90,7 +90,7 @@ class CollectionViewSet(BaseViewSet): class CollectionItemViewSet(BaseViewSet): - allowed_methods = ['GET', 'POST'] + allowed_methods = ['GET', 'POST', 'PUT'] permission_classes = BaseViewSet.permission_classes queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer @@ -133,11 +133,7 @@ class CollectionItemViewSet(BaseViewSet): return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) def destroy(self, request, collection_uid=None, uid=None): - # FIXME: implement - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def update(self, request, collection_uid=None, uid=None): - # FIXME: implement, or should it be implemented elsewhere? + # We can't have destroy because we need to get data from the user (in the body) such as hmac. return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def partial_update(self, request, collection_uid=None, uid=None): From 452a8f1e7effef4db934d0513b5529d46f44ec81 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 16:07:55 +0200 Subject: [PATCH 069/511] Implement item creation. --- django_etesync/serializers.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index b3f7254..a576b6d 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -118,6 +118,20 @@ class CollectionItemSerializer(serializers.ModelSerializer): model = models.CollectionItem fields = ('uid', 'content') + def create(self, validated_data): + """Function that's called when this serializer creates an item""" + revision_data = validated_data.pop('content') + instance = self.__class__.Meta.model(**validated_data) + + with transaction.atomic(): + instance.save() + + chunks = revision_data.pop('chunks') + revision = models.CollectionItemRevision.objects.create(**revision_data, item=instance) + revision.chunks.set(chunks) + + return instance + def update(self, instance, validated_data): """Function that's called when this serializer is meant to update an item""" revision_data = validated_data.pop('content') From f4cb7cb74f2252ae46268dc3195048f24bc2cdbd Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 16:42:49 +0200 Subject: [PATCH 070/511] Collection item list: limit only to non-deleted by default. --- django_etesync/views.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index b90b62e..b1bb18b 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -110,7 +110,9 @@ class CollectionItemViewSet(BaseViewSet): except Collection.DoesNotExist: raise Http404("Collection does not exist") # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') - queryset = type(self).queryset.filter(collection__pk=collection.pk) + queryset = type(self).queryset.filter(collection__pk=collection.pk, + revisions__current=True, + revisions__isDeletion=False) return queryset From f1bfb0a9a057aab754b93dab32e0f442e6c6d474 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 16:46:41 +0200 Subject: [PATCH 071/511] Model uid validator: fix off-by-1 error with the uid. 256bit is actually 43 base64 chars, not 44. --- .../migrations/0016_auto_20200226_1446.py | 29 +++++++++++++++++++ django_etesync/models.py | 2 +- 2 files changed, 30 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0016_auto_20200226_1446.py diff --git a/django_etesync/migrations/0016_auto_20200226_1446.py b/django_etesync/migrations/0016_auto_20200226_1446.py new file mode 100644 index 0000000..2929cbf --- /dev/null +++ b/django_etesync/migrations/0016_auto_20200226_1446.py @@ -0,0 +1,29 @@ +# Generated by Django 3.0.3 on 2020-02-26 14:46 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0015_auto_20200226_1349'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{43}')]), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{43}')]), + ), + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{43}')]), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index dda081f..4577efb 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -20,7 +20,7 @@ from django.core.validators import RegexValidator from django.utils.functional import cached_property -UidValidator = RegexValidator(regex=r'[a-zA-Z0-9\-_=]{44}', message='Not a valid UID. Expected a 256bit base64url.') +UidValidator = RegexValidator(regex=r'[a-zA-Z0-9\-_=]{43}', message='Not a valid UID. Expected a 256bit base64url.') class Collection(models.Model): From 0ee00e1a9fff8e3b7a2fd0855b3b12665614a7e8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 16:55:47 +0200 Subject: [PATCH 072/511] Collection item: rename isDeletion to deleted --- .../migrations/0017_auto_20200226_1455.py | 18 ++++++++++++++++++ django_etesync/models.py | 2 +- django_etesync/serializers.py | 2 +- django_etesync/views.py | 2 +- 4 files changed, 21 insertions(+), 3 deletions(-) create mode 100644 django_etesync/migrations/0017_auto_20200226_1455.py diff --git a/django_etesync/migrations/0017_auto_20200226_1455.py b/django_etesync/migrations/0017_auto_20200226_1455.py new file mode 100644 index 0000000..2148123 --- /dev/null +++ b/django_etesync/migrations/0017_auto_20200226_1455.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-26 14:55 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0016_auto_20200226_1446'), + ] + + operations = [ + migrations.RenameField( + model_name='collectionitemrevision', + old_name='isDeletion', + new_name='deleted', + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 4577efb..1792031 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -81,7 +81,7 @@ class CollectionItemRevision(models.Model): chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, blank=True, null=True) - isDeletion = models.BooleanField(default=False) + deleted = models.BooleanField(default=False) class Meta: unique_together = ('item', 'current') diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index a576b6d..8736a02 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -75,7 +75,7 @@ class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemRevision - fields = ('version', 'encryptionKey', 'chunks', 'hmac', 'isDeletion') + fields = ('version', 'encryptionKey', 'chunks', 'hmac', 'deleted') class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): diff --git a/django_etesync/views.py b/django_etesync/views.py index b1bb18b..3d27e72 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -112,7 +112,7 @@ class CollectionItemViewSet(BaseViewSet): # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = type(self).queryset.filter(collection__pk=collection.pk, revisions__current=True, - revisions__isDeletion=False) + revisions__deleted=False) return queryset From e0d593a9b6b8df6497f0a9c0fc222961a32a960c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 20:04:26 +0200 Subject: [PATCH 073/511] Collection Item Revision: dissalow blank for the current field. --- .../migrations/0018_auto_20200226_1803.py | 18 ++++++++++++++++++ django_etesync/models.py | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0018_auto_20200226_1803.py diff --git a/django_etesync/migrations/0018_auto_20200226_1803.py b/django_etesync/migrations/0018_auto_20200226_1803.py new file mode 100644 index 0000000..ae9200d --- /dev/null +++ b/django_etesync/migrations/0018_auto_20200226_1803.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-02-26 18:03 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0017_auto_20200226_1455'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemrevision', + name='current', + field=models.BooleanField(db_index=True, default=True, null=True), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 1792031..b11c36c 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -80,7 +80,7 @@ class CollectionItemRevision(models.Model): item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) - current = models.BooleanField(db_index=True, default=True, blank=True, null=True) + current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) class Meta: From be11e3e0e6e198460165a6466dd0a21c3c125956 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 20:38:07 +0200 Subject: [PATCH 074/511] Collection: implement collection membership. --- .../migrations/0019_collectionmember.py | 29 +++++++++++++++++++ django_etesync/models.py | 22 ++++++++++++++ django_etesync/serializers.py | 16 ++++------ 3 files changed, 57 insertions(+), 10 deletions(-) create mode 100644 django_etesync/migrations/0019_collectionmember.py diff --git a/django_etesync/migrations/0019_collectionmember.py b/django_etesync/migrations/0019_collectionmember.py new file mode 100644 index 0000000..142e945 --- /dev/null +++ b/django_etesync/migrations/0019_collectionmember.py @@ -0,0 +1,29 @@ +# Generated by Django 3.0.3 on 2020-02-26 18:33 + +from django.conf import settings +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ('django_etesync', '0018_auto_20200226_1803'), + ] + + operations = [ + migrations.CreateModel( + name='CollectionMember', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('encryptionKey', models.BinaryField(editable=True)), + ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='members', to='django_etesync.Collection')), + ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ], + options={ + 'unique_together': {('user', 'collection')}, + }, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index b11c36c..3be7b06 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -88,3 +88,25 @@ class CollectionItemRevision(models.Model): def __str__(self): return '{} {} current={}'.format(self.item.uid, self.id, self.current) + + +class CollectionMember(models.Model): + class AccessLevels(models.TextChoices): + ADMIN = 'adm' + READ_WRITE = 'rw' + READ_ONLY = 'ro' + + collection = models.ForeignKey(Collection, related_name='members', on_delete=models.CASCADE) + user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + encryptionKey = models.BinaryField(editable=True, blank=False, null=False) + accessLevel = models.CharField( + max_length=3, + choices=AccessLevels.choices, + default=AccessLevels.READ_ONLY, + ) + + class Meta: + unique_together = ('user', 'collection') + + def __str__(self): + return '{} {}'.format(self.collection.uid, self.user) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 8736a02..c194243 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -31,29 +31,25 @@ class BinaryBase64Field(serializers.Field): class CollectionSerializer(serializers.ModelSerializer): - owner = serializers.SlugRelatedField( - slug_field=User.USERNAME_FIELD, - read_only=True - ) encryptionKey = serializers.SerializerMethodField('get_key_from_context') - permissions = serializers.SerializerMethodField('get_permission_from_context') + accessLevel = serializers.SerializerMethodField('get_access_level_from_context') ctag = serializers.SerializerMethodField('get_ctag') class Meta: model = models.Collection - fields = ('uid', 'version', 'owner', 'encryptionKey', 'permissions', 'ctag') + fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'ctag') def get_key_from_context(self, obj): request = self.context.get('request', None) if request is not None: - return 'FIXME' + return obj.members.get(user=request.user).encryptionKey return None - def get_permission_from_context(self, obj): + def get_access_level_from_context(self, obj): request = self.context.get('request', None) if request is not None: - return 'FIXME' - return 'readOnly' + return obj.members.get(user=request.user).accessLevel + return None def get_ctag(self, obj): return 'FIXME' From 3eb79e0a04444f3b3ec5b3e4110a1b9910299c81 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 20:42:28 +0200 Subject: [PATCH 075/511] Create collection member when creating collection. --- django_etesync/views.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 3d27e72..60e79a6 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -24,7 +24,7 @@ from rest_framework.decorators import action as action_decorator from rest_framework.response import Response from . import app_settings, paginators -from .models import Collection, CollectionItem +from .models import Collection, CollectionItem, CollectionMember from .serializers import ( CollectionSerializer, CollectionItemSerializer, @@ -73,7 +73,12 @@ class CollectionViewSet(BaseViewSet): if serializer.is_valid(): try: with transaction.atomic(): - serializer.save(owner=self.request.user) + col = serializer.save(owner=self.request.user) + CollectionMember(collection=col, + user=self.request.user, + accessLevel=CollectionMember.AccessLevels.ADMIN, + encryptionKey=serializer.validated_data['encryptionKey'] + ).save() except IntegrityError: content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) From 7a38e26872a9d0094e056ebe67862be3867187a0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 20:54:00 +0200 Subject: [PATCH 076/511] Collection: fix issue with encryptionKey not being base64 encoded. --- django_etesync/serializers.py | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index c194243..5a57900 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -30,8 +30,16 @@ class BinaryBase64Field(serializers.Field): return base64.b64decode(data) +class CollectionEncryptionKeyField(BinaryBase64Field): + def get_attribute(self, instance): + request = self.context.get('request', None) + if request is not None: + return instance.members.get(user=request.user).encryptionKey + return None + + class CollectionSerializer(serializers.ModelSerializer): - encryptionKey = serializers.SerializerMethodField('get_key_from_context') + encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') ctag = serializers.SerializerMethodField('get_ctag') @@ -39,12 +47,6 @@ class CollectionSerializer(serializers.ModelSerializer): model = models.Collection fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'ctag') - def get_key_from_context(self, obj): - request = self.context.get('request', None) - if request is not None: - return obj.members.get(user=request.user).encryptionKey - return None - def get_access_level_from_context(self, obj): request = self.context.get('request', None) if request is not None: From 771d2d013deb5891389a6901b3ce4635d73d1f3a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 21:11:29 +0200 Subject: [PATCH 077/511] Fix creation of collection membership when creating collections. --- django_etesync/serializers.py | 16 ++++++++++++++++ django_etesync/views.py | 12 +++--------- 2 files changed, 19 insertions(+), 9 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 5a57900..4d4558b 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -56,6 +56,22 @@ class CollectionSerializer(serializers.ModelSerializer): def get_ctag(self, obj): return 'FIXME' + def create(self, validated_data): + """Function that's called when this serializer creates an item""" + encryption_key = validated_data.pop('encryptionKey') + instance = self.__class__.Meta.model(**validated_data) + + print(validated_data) + with transaction.atomic(): + instance.save() + models.CollectionMember(collection=instance, + user=validated_data.get('owner'), + accessLevel=models.CollectionMember.AccessLevels.ADMIN, + encryptionKey=encryption_key, + ).save() + + return instance + class CollectionItemChunkSerializer(serializers.ModelSerializer): class Meta: diff --git a/django_etesync/views.py b/django_etesync/views.py index 60e79a6..9bdb244 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -13,7 +13,7 @@ # along with this program. If not, see . from django.contrib.auth import get_user_model -from django.db import IntegrityError, transaction +from django.db import IntegrityError from django.http import Http404 from django.shortcuts import get_object_or_404 @@ -24,7 +24,7 @@ from rest_framework.decorators import action as action_decorator from rest_framework.response import Response from . import app_settings, paginators -from .models import Collection, CollectionItem, CollectionMember +from .models import Collection, CollectionItem from .serializers import ( CollectionSerializer, CollectionItemSerializer, @@ -72,13 +72,7 @@ class CollectionViewSet(BaseViewSet): serializer = self.serializer_class(data=request.data) if serializer.is_valid(): try: - with transaction.atomic(): - col = serializer.save(owner=self.request.user) - CollectionMember(collection=col, - user=self.request.user, - accessLevel=CollectionMember.AccessLevels.ADMIN, - encryptionKey=serializer.validated_data['encryptionKey'] - ).save() + serializer.save(owner=self.request.user) except IntegrityError: content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) From c74ed50bd5029aebc31841912ca573af03b97aa2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 21:13:33 +0200 Subject: [PATCH 078/511] Collection: filter queryset only to collections for which the user has access to. --- django_etesync/views.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 9bdb244..8de7313 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -50,7 +50,8 @@ class BaseViewSet(viewsets.ModelViewSet): return serializer_class def get_collection_queryset(self, queryset=Collection.objects): - return queryset.all() + user = self.request.user + return queryset.filter(members__user=user) class CollectionViewSet(BaseViewSet): @@ -143,7 +144,7 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): - col = get_object_or_404(Collection.objects, uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) col_it = get_object_or_404(col.items, uid=uid) serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) @@ -169,7 +170,8 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): lookup_field = 'uid' def get_collection_queryset(self, queryset=Collection.objects): - return queryset.all() + user = self.request.user + return queryset.filter(members__user=user) def create(self, request, collection_uid=None, collection_item_uid=None): col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) From 5ceaa9fb1ab288488b343ba0585644bdc5cdc510 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 26 Feb 2020 21:22:58 +0200 Subject: [PATCH 079/511] Collection: calculate a value for ctag in the meanwhile. --- django_etesync/serializers.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 4d4558b..f3e0fbd 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -54,7 +54,13 @@ class CollectionSerializer(serializers.ModelSerializer): return None def get_ctag(self, obj): - return 'FIXME' + # FIXME: we need to have something that's more privacy friendly + last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() + if last_revision is None: + # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. + return None + + return str(last_revision.id) def create(self, validated_data): """Function that's called when this serializer creates an item""" From 3beb7ac4bbf335d2b4f69568828e8d38eceade29 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 10 Mar 2020 16:27:57 +0200 Subject: [PATCH 080/511] Requirements: add pywatchman for more efficient watching. --- requirements.in/development.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements.in/development.txt b/requirements.in/development.txt index 30fb558..c752bfb 100644 --- a/requirements.in/development.txt +++ b/requirements.in/development.txt @@ -1,2 +1,3 @@ coverage pip-tools +pywatchman From d587f8185bf046cd6d0bf86dd4da26dbe44b4b3f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 10 Mar 2020 16:40:42 +0200 Subject: [PATCH 081/511] Uids: change uids to be much shorter and base62 for non-chunks. --- .../migrations/0020_auto_20200310_1438.py | 29 +++++++++++++++++++ .../migrations/0021_auto_20200310_1439.py | 24 +++++++++++++++ django_etesync/models.py | 5 ++-- 3 files changed, 56 insertions(+), 2 deletions(-) create mode 100644 django_etesync/migrations/0020_auto_20200310_1438.py create mode 100644 django_etesync/migrations/0021_auto_20200310_1439.py diff --git a/django_etesync/migrations/0020_auto_20200310_1438.py b/django_etesync/migrations/0020_auto_20200310_1438.py new file mode 100644 index 0000000..6949145 --- /dev/null +++ b/django_etesync/migrations/0020_auto_20200310_1438.py @@ -0,0 +1,29 @@ +# Generated by Django 3.0.3 on 2020-03-10 14:38 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0019_collectionmember'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]{24}')]), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]{24}')]), + ), + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}=?$')]), + ), + ] diff --git a/django_etesync/migrations/0021_auto_20200310_1439.py b/django_etesync/migrations/0021_auto_20200310_1439.py new file mode 100644 index 0000000..3f1341e --- /dev/null +++ b/django_etesync/migrations/0021_auto_20200310_1439.py @@ -0,0 +1,24 @@ +# Generated by Django 3.0.3 on 2020-03-10 14:39 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0020_auto_20200310_1438'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 3be7b06..7eebd1a 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -20,7 +20,8 @@ from django.core.validators import RegexValidator from django.utils.functional import cached_property -UidValidator = RegexValidator(regex=r'[a-zA-Z0-9\-_=]{43}', message='Not a valid UID. Expected a 256bit base64url.') +Base64Url256BitValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{43}=?$', message='Expected a 256bit base64url.') +UidValidator = RegexValidator(regex=r'[a-zA-Z0-9]', message='Not a valid UID') class Collection(models.Model): @@ -61,7 +62,7 @@ def chunk_directory_path(instance, filename): class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, - max_length=44, validators=[UidValidator]) + max_length=44, validators=[Base64Url256BitValidator]) item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) order = models.CharField(max_length=100, blank=False, null=False) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) From dfbfa01bc5e7ecb45f771bfbc14cc90e8e85e8b6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 10 Mar 2020 17:49:23 +0200 Subject: [PATCH 082/511] CollectionItem: move version and encryption key to the item itself. --- .../migrations/0022_auto_20200310_1547.py | 33 +++++++++++++++++++ django_etesync/models.py | 4 +-- django_etesync/serializers.py | 9 ++--- 3 files changed, 40 insertions(+), 6 deletions(-) create mode 100644 django_etesync/migrations/0022_auto_20200310_1547.py diff --git a/django_etesync/migrations/0022_auto_20200310_1547.py b/django_etesync/migrations/0022_auto_20200310_1547.py new file mode 100644 index 0000000..cbd0ee7 --- /dev/null +++ b/django_etesync/migrations/0022_auto_20200310_1547.py @@ -0,0 +1,33 @@ +# Generated by Django 3.0.3 on 2020-03-10 15:47 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0021_auto_20200310_1439'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitemrevision', + name='encryptionKey', + ), + migrations.RemoveField( + model_name='collectionitemrevision', + name='version', + ), + migrations.AddField( + model_name='collectionitem', + name='encryptionKey', + field=models.BinaryField(default=b'aoesnutheounth', editable=True), + preserve_default=False, + ), + migrations.AddField( + model_name='collectionitem', + name='version', + field=models.PositiveSmallIntegerField(default=1), + preserve_default=False, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 7eebd1a..d996769 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -41,6 +41,8 @@ class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[UidValidator]) collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) + version = models.PositiveSmallIntegerField() + encryptionKey = models.BinaryField(editable=True, blank=False, null=False) class Meta: unique_together = ('uid', 'collection') @@ -76,8 +78,6 @@ class CollectionItemChunk(models.Model): class CollectionItemRevision(models.Model): - version = models.PositiveSmallIntegerField() - encryptionKey = models.BinaryField(editable=True, blank=False, null=False) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index f3e0fbd..c24239b 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -54,7 +54,8 @@ class CollectionSerializer(serializers.ModelSerializer): return None def get_ctag(self, obj): - # FIXME: we need to have something that's more privacy friendly + # FIXME: we need to have something that's more privacy friendly. Can probably just generate a uid per revision + # on revision creation (on the server) and just use that. last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() if last_revision is None: # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. @@ -86,7 +87,6 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): - encryptionKey = BinaryBase64Field() chunks = serializers.SlugRelatedField( slug_field='uid', queryset=models.CollectionItemChunk.objects.all(), @@ -95,7 +95,7 @@ class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemRevision - fields = ('version', 'encryptionKey', 'chunks', 'hmac', 'deleted') + fields = ('chunks', 'hmac', 'deleted') class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): @@ -132,11 +132,12 @@ class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerialize class CollectionItemSerializer(serializers.ModelSerializer): + encryptionKey = BinaryBase64Field() content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.CollectionItem - fields = ('uid', 'content') + fields = ('uid', 'version', 'encryptionKey', 'content') def create(self, validated_data): """Function that's called when this serializer creates an item""" From 23edc29bb81c1ab6671fb43f57c19b1a28543767 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 10 Mar 2020 17:56:24 +0200 Subject: [PATCH 083/511] Chunks: order based on item too so items are clustered together. --- .../migrations/0023_auto_20200310_1556.py | 17 +++++++++++++++++ django_etesync/models.py | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0023_auto_20200310_1556.py diff --git a/django_etesync/migrations/0023_auto_20200310_1556.py b/django_etesync/migrations/0023_auto_20200310_1556.py new file mode 100644 index 0000000..e2a9b80 --- /dev/null +++ b/django_etesync/migrations/0023_auto_20200310_1556.py @@ -0,0 +1,17 @@ +# Generated by Django 3.0.3 on 2020-03-10 15:56 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0022_auto_20200310_1547'), + ] + + operations = [ + migrations.AlterModelOptions( + name='collectionitemchunk', + options={'ordering': ('item', 'order')}, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index d996769..df25867 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -71,7 +71,7 @@ class CollectionItemChunk(models.Model): class Meta: unique_together = ('item', 'order') - ordering = ['order'] + ordering = ('item', 'order') def __str__(self): return self.uid From f8a94eeb04bec54e92c753aeff11dbed51ae7e6d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 12 Mar 2020 15:52:36 +0200 Subject: [PATCH 084/511] Revision: add a proper uid for revisions (which we also use for sync tag). --- .../0024_collectionitemrevision_uid.py | 19 +++++++++++++++++++ .../migrations/0025_auto_20200312_1350.py | 19 +++++++++++++++++++ django_etesync/models.py | 4 +++- django_etesync/serializers.py | 15 ++++++++++----- 4 files changed, 51 insertions(+), 6 deletions(-) create mode 100644 django_etesync/migrations/0024_collectionitemrevision_uid.py create mode 100644 django_etesync/migrations/0025_auto_20200312_1350.py diff --git a/django_etesync/migrations/0024_collectionitemrevision_uid.py b/django_etesync/migrations/0024_collectionitemrevision_uid.py new file mode 100644 index 0000000..6134c89 --- /dev/null +++ b/django_etesync/migrations/0024_collectionitemrevision_uid.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-03-12 13:41 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0023_auto_20200310_1556'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemrevision', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), + ), + ] diff --git a/django_etesync/migrations/0025_auto_20200312_1350.py b/django_etesync/migrations/0025_auto_20200312_1350.py new file mode 100644 index 0000000..b54aeff --- /dev/null +++ b/django_etesync/migrations/0025_auto_20200312_1350.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-03-12 13:50 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0024_collectionitemrevision_uid'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemrevision', + name='uid', + field=models.CharField(db_index=True, max_length=44, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index df25867..8079c28 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -78,6 +78,8 @@ class CollectionItemChunk(models.Model): class CollectionItemRevision(models.Model): + uid = models.CharField(db_index=True, unique=True, blank=False, null=False, + max_length=44, validators=[UidValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) @@ -88,7 +90,7 @@ class CollectionItemRevision(models.Model): unique_together = ('item', 'current') def __str__(self): - return '{} {} current={}'.format(self.item.uid, self.id, self.current) + return '{} {} current={}'.format(self.uid, self.item.uid, self.current) class CollectionMember(models.Model): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index c24239b..c6d3a86 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -16,12 +16,17 @@ import base64 from django.contrib.auth import get_user_model from django.db import transaction +from django.utils.crypto import get_random_string from rest_framework import serializers from . import models User = get_user_model() +def generate_rev_uid(length=32): + return get_random_string(length) + + class BinaryBase64Field(serializers.Field): def to_representation(self, value): return base64.b64encode(value).decode('ascii') @@ -54,14 +59,12 @@ class CollectionSerializer(serializers.ModelSerializer): return None def get_ctag(self, obj): - # FIXME: we need to have something that's more privacy friendly. Can probably just generate a uid per revision - # on revision creation (on the server) and just use that. last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() if last_revision is None: # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. return None - return str(last_revision.id) + return last_revision.uid def create(self, validated_data): """Function that's called when this serializer creates an item""" @@ -148,7 +151,8 @@ class CollectionItemSerializer(serializers.ModelSerializer): instance.save() chunks = revision_data.pop('chunks') - revision = models.CollectionItemRevision.objects.create(**revision_data, item=instance) + revision = models.CollectionItemRevision.objects.create(**revision_data, uid=generate_rev_uid(), + item=instance) revision.chunks.set(chunks) return instance @@ -165,7 +169,8 @@ class CollectionItemSerializer(serializers.ModelSerializer): current_revision.save() chunks = revision_data.pop('chunks') - revision = models.CollectionItemRevision.objects.create(**revision_data, item=instance) + revision = models.CollectionItemRevision.objects.create(**revision_data, uid=generate_rev_uid(), + item=instance) revision.chunks.set(chunks) return instance From d1df6db8b16d3a544568b344ca70fffaa2e2c994 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 12 Mar 2020 16:02:00 +0200 Subject: [PATCH 085/511] Revision: add metadata field. --- .../0026_collectionitemrevision_meta.py | 18 ++++++++++++++++++ django_etesync/models.py | 1 + 2 files changed, 19 insertions(+) create mode 100644 django_etesync/migrations/0026_collectionitemrevision_meta.py diff --git a/django_etesync/migrations/0026_collectionitemrevision_meta.py b/django_etesync/migrations/0026_collectionitemrevision_meta.py new file mode 100644 index 0000000..8056e61 --- /dev/null +++ b/django_etesync/migrations/0026_collectionitemrevision_meta.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-03-12 14:01 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0025_auto_20200312_1350'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemrevision', + name='meta', + field=models.BinaryField(blank=True, editable=True, null=True), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 8079c28..02a173c 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -81,6 +81,7 @@ class CollectionItemRevision(models.Model): uid = models.CharField(db_index=True, unique=True, blank=False, null=False, max_length=44, validators=[UidValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) + meta = models.BinaryField(editable=True, blank=True, null=True) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') hmac = models.CharField(max_length=50, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) From c56cbb3f8209bd79c4ac8e0273c54e43a41f064e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 12 Mar 2020 16:06:15 +0200 Subject: [PATCH 086/511] Remove debug print. --- django_etesync/serializers.py | 1 - 1 file changed, 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index c6d3a86..2c12592 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -71,7 +71,6 @@ class CollectionSerializer(serializers.ModelSerializer): encryption_key = validated_data.pop('encryptionKey') instance = self.__class__.Meta.model(**validated_data) - print(validated_data) with transaction.atomic(): instance.save() models.CollectionMember(collection=instance, From 66e5062461398b0cbec6f4fa28b73f92577fe73e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 12 Mar 2020 21:02:27 +0200 Subject: [PATCH 087/511] Collection: add content support. --- .../migrations/0027_collection_mainitem.py | 19 ++++ .../migrations/0028_auto_20200312_1819.py | 24 +++++ .../migrations/0029_auto_20200312_1849.py | 31 +++++++ .../migrations/0030_auto_20200312_1859.py | 19 ++++ django_etesync/models.py | 11 ++- django_etesync/serializers.py | 87 +++++++++++-------- 6 files changed, 154 insertions(+), 37 deletions(-) create mode 100644 django_etesync/migrations/0027_collection_mainitem.py create mode 100644 django_etesync/migrations/0028_auto_20200312_1819.py create mode 100644 django_etesync/migrations/0029_auto_20200312_1849.py create mode 100644 django_etesync/migrations/0030_auto_20200312_1859.py diff --git a/django_etesync/migrations/0027_collection_mainitem.py b/django_etesync/migrations/0027_collection_mainitem.py new file mode 100644 index 0000000..b420d8f --- /dev/null +++ b/django_etesync/migrations/0027_collection_mainitem.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-03-12 14:14 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0026_collectionitemrevision_meta'), + ] + + operations = [ + migrations.AddField( + model_name='collection', + name='mainItem', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, related_name='of_collection', to='django_etesync.CollectionItem'), + ), + ] diff --git a/django_etesync/migrations/0028_auto_20200312_1819.py b/django_etesync/migrations/0028_auto_20200312_1819.py new file mode 100644 index 0000000..6d76499 --- /dev/null +++ b/django_etesync/migrations/0028_auto_20200312_1819.py @@ -0,0 +1,24 @@ +# Generated by Django 3.0.3 on 2020-03-12 18:19 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0027_collection_mainitem'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitem', + name='encryptionKey', + field=models.BinaryField(editable=True, null=True), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=44, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), + ), + ] diff --git a/django_etesync/migrations/0029_auto_20200312_1849.py b/django_etesync/migrations/0029_auto_20200312_1849.py new file mode 100644 index 0000000..165b405 --- /dev/null +++ b/django_etesync/migrations/0029_auto_20200312_1849.py @@ -0,0 +1,31 @@ +# Generated by Django 3.0.3 on 2020-03-12 18:49 + +from django.db import migrations +from django_etesync.serializers import generate_rev_uid + + +def add_collection_main_item(apps, schema_editor): + Collection = apps.get_model('django_etesync', 'Collection') + CollectionItem = apps.get_model('django_etesync', 'CollectionItem') + CollectionItemRevision = apps.get_model('django_etesync', 'CollectionItemRevision') + + for col in Collection.objects.all(): + main_item = CollectionItem.objects.create(uid=None, encryptionKey=None, version=col.version, collection=col) + col.mainItem = main_item + col.save() + + CollectionItemRevision.objects.create( + uid=generate_rev_uid(), + hmac='hmac-hash', + item=main_item) + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0028_auto_20200312_1819'), + ] + + operations = [ + migrations.RunPython(add_collection_main_item), + ] diff --git a/django_etesync/migrations/0030_auto_20200312_1859.py b/django_etesync/migrations/0030_auto_20200312_1859.py new file mode 100644 index 0000000..fe8050a --- /dev/null +++ b/django_etesync/migrations/0030_auto_20200312_1859.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-03-12 18:59 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0029_auto_20200312_1849'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='mainItem', + field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, related_name='of_collection', to='django_etesync.CollectionItem'), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 02a173c..6c30b00 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -29,6 +29,7 @@ class Collection(models.Model): max_length=44, validators=[UidValidator]) version = models.PositiveSmallIntegerField() owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + mainItem = models.OneToOneField('CollectionItem', related_name='of_collection', on_delete=models.PROTECT) class Meta: unique_together = ('uid', 'owner') @@ -36,19 +37,23 @@ class Collection(models.Model): def __str__(self): return self.uid + @cached_property + def content(self): + return self.mainItem.content + class CollectionItem(models.Model): - uid = models.CharField(db_index=True, blank=False, null=False, + uid = models.CharField(db_index=True, blank=False, null=True, max_length=44, validators=[UidValidator]) collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) version = models.PositiveSmallIntegerField() - encryptionKey = models.BinaryField(editable=True, blank=False, null=False) + encryptionKey = models.BinaryField(editable=True, blank=False, null=True) class Meta: unique_together = ('uid', 'collection') def __str__(self): - return self.uid + return '{} {}'.format(self.uid, self.collection.uid) @cached_property def content(self): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 2c12592..7a2d3fd 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -43,44 +43,13 @@ class CollectionEncryptionKeyField(BinaryBase64Field): return None -class CollectionSerializer(serializers.ModelSerializer): - encryptionKey = CollectionEncryptionKeyField() - accessLevel = serializers.SerializerMethodField('get_access_level_from_context') - ctag = serializers.SerializerMethodField('get_ctag') - - class Meta: - model = models.Collection - fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'ctag') - - def get_access_level_from_context(self, obj): +class CollectionContentField(BinaryBase64Field): + def get_attribute(self, instance): request = self.context.get('request', None) if request is not None: - return obj.members.get(user=request.user).accessLevel + return instance.members.get(user=request.user).encryptionKey return None - def get_ctag(self, obj): - last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() - if last_revision is None: - # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. - return None - - return last_revision.uid - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - encryption_key = validated_data.pop('encryptionKey') - instance = self.__class__.Meta.model(**validated_data) - - with transaction.atomic(): - instance.save() - models.CollectionMember(collection=instance, - user=validated_data.get('owner'), - accessLevel=models.CollectionMember.AccessLevels.ADMIN, - encryptionKey=encryption_key, - ).save() - - return instance - class CollectionItemChunkSerializer(serializers.ModelSerializer): class Meta: @@ -177,3 +146,53 @@ class CollectionItemSerializer(serializers.ModelSerializer): class CollectionItemInlineSerializer(CollectionItemSerializer): content = CollectionItemRevisionInlineSerializer(read_only=True, many=False) + + +class CollectionSerializer(serializers.ModelSerializer): + encryptionKey = CollectionEncryptionKeyField() + accessLevel = serializers.SerializerMethodField('get_access_level_from_context') + ctag = serializers.SerializerMethodField('get_ctag') + content = CollectionItemRevisionSerializer(many=False) + + class Meta: + model = models.Collection + fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'ctag') + + def get_access_level_from_context(self, obj): + request = self.context.get('request', None) + if request is not None: + return obj.members.get(user=request.user).accessLevel + return None + + def get_ctag(self, obj): + last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() + if last_revision is None: + # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. + return None + + return last_revision.uid + + def create(self, validated_data): + """Function that's called when this serializer creates an item""" + revision_data = validated_data.pop('content') + encryption_key = validated_data.pop('encryptionKey') + instance = self.__class__.Meta.model(**validated_data) + + with transaction.atomic(): + main_item = models.CollectionItem.objects.create( + uid=None, encryptionKey=None, version=instance.version, collection=instance) + instance.mainItem = main_item + + chunks = revision_data.pop('chunks') + revision = models.CollectionItemRevision.objects.create(**revision_data, uid=generate_rev_uid(), + item=main_item) + revision.chunks.set(chunks) + + instance.save() + models.CollectionMember(collection=instance, + user=validated_data.get('owner'), + accessLevel=models.CollectionMember.AccessLevels.ADMIN, + encryptionKey=encryption_key, + ).save() + + return instance From 86b6a449176bbfeda363d25b467777cb3013729a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 17 Mar 2020 17:10:53 +0200 Subject: [PATCH 088/511] We use base64url without padding. --- .../migrations/0031_auto_20200317_1509.py | 19 +++++++++++++++++++ django_etesync/models.py | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0031_auto_20200317_1509.py diff --git a/django_etesync/migrations/0031_auto_20200317_1509.py b/django_etesync/migrations/0031_auto_20200317_1509.py new file mode 100644 index 0000000..7166781 --- /dev/null +++ b/django_etesync/migrations/0031_auto_20200317_1509.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-03-17 15:09 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0030_auto_20200312_1859'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')]), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 6c30b00..8df5d44 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -20,7 +20,7 @@ from django.core.validators import RegexValidator from django.utils.functional import cached_property -Base64Url256BitValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{43}=?$', message='Expected a 256bit base64url.') +Base64Url256BitValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{43}$', message='Expected a 256bit base64url.') UidValidator = RegexValidator(regex=r'[a-zA-Z0-9]', message='Not a valid UID') From ab86a912cd8d0ecb3dbe071a9ad53a5a984e626a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 17 Mar 2020 17:13:47 +0200 Subject: [PATCH 089/511] Revision uid is now the hmac. --- .../migrations/0032_auto_20200317_1513.py | 23 +++++++++++++++++++ django_etesync/models.py | 3 +-- django_etesync/serializers.py | 2 +- 3 files changed, 25 insertions(+), 3 deletions(-) create mode 100644 django_etesync/migrations/0032_auto_20200317_1513.py diff --git a/django_etesync/migrations/0032_auto_20200317_1513.py b/django_etesync/migrations/0032_auto_20200317_1513.py new file mode 100644 index 0000000..0546711 --- /dev/null +++ b/django_etesync/migrations/0032_auto_20200317_1513.py @@ -0,0 +1,23 @@ +# Generated by Django 3.0.3 on 2020-03-17 15:13 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0031_auto_20200317_1509'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitemrevision', + name='hmac', + ), + migrations.AlterField( + model_name='collectionitemrevision', + name='uid', + field=models.CharField(db_index=True, max_length=44, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')]), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 8df5d44..586c105 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -84,11 +84,10 @@ class CollectionItemChunk(models.Model): class CollectionItemRevision(models.Model): uid = models.CharField(db_index=True, unique=True, blank=False, null=False, - max_length=44, validators=[UidValidator]) + max_length=44, validators=[Base64Url256BitValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=True, null=True) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') - hmac = models.CharField(max_length=50, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 7a2d3fd..33a09ac 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -66,7 +66,7 @@ class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'hmac', 'deleted') + fields = ('chunks', 'uid', 'deleted') class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): From 80ef568397bf790fae6f7a77311c23cff184f1a3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 17 Mar 2020 22:10:33 +0200 Subject: [PATCH 090/511] Make meta not-null. --- .../migrations/0033_auto_20200317_2010.py | 19 +++++++++++++++++++ django_etesync/models.py | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0033_auto_20200317_2010.py diff --git a/django_etesync/migrations/0033_auto_20200317_2010.py b/django_etesync/migrations/0033_auto_20200317_2010.py new file mode 100644 index 0000000..7a42b38 --- /dev/null +++ b/django_etesync/migrations/0033_auto_20200317_2010.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-03-17 20:10 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0032_auto_20200317_1513'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemrevision', + name='meta', + field=models.BinaryField(default=b'', editable=True), + preserve_default=False, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 586c105..ca3dea9 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -86,7 +86,7 @@ class CollectionItemRevision(models.Model): uid = models.CharField(db_index=True, unique=True, blank=False, null=False, max_length=44, validators=[Base64Url256BitValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) - meta = models.BinaryField(editable=True, blank=True, null=True) + meta = models.BinaryField(editable=True, blank=False, null=False) chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) From 2ac0b55de971dba99e2c0355687a4298761a36e9 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 17 Mar 2020 22:11:18 +0200 Subject: [PATCH 091/511] Revision: expose meta. --- django_etesync/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 33a09ac..127d88c 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -66,7 +66,7 @@ class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'uid', 'deleted') + fields = ('chunks', 'meta', 'uid', 'deleted') class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): From 73a55b98171d865752fd02680f66d365a128e48a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 26 Mar 2020 18:32:34 +0200 Subject: [PATCH 092/511] Add reset url so it can be used with client test suites. --- etesync_server/urls.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/etesync_server/urls.py b/etesync_server/urls.py index 5f3a6d7..6a079de 100644 --- a/etesync_server/urls.py +++ b/etesync_server/urls.py @@ -27,6 +27,7 @@ Including another URLconf 1. Import the include() function: from django.urls import include, path 2. Add a URL to urlpatterns: path('blog/', include('blog.urls')) """ +from django.conf import settings from django.urls import include, path, re_path from django.contrib import admin from django.views.generic import TemplateView @@ -54,3 +55,8 @@ urlpatterns = [ path('admin/', admin.site.urls), path('', TemplateView.as_view(template_name='success.html')), ] + +if settings.DEBUG: + urlpatterns = [ + path('reset/', views.reset, name='reset_debug'), + ] + urlpatterns From 9b13404ce74c8953432759e83395a1bffcf4e42d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 14 Apr 2020 16:21:51 +0300 Subject: [PATCH 093/511] Add a reset view for tests. --- django_etesync/views.py | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 8de7313..a4b8108 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -12,9 +12,10 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . +from django.conf import settings from django.contrib.auth import get_user_model from django.db import IntegrityError -from django.http import Http404 +from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 from rest_framework import status @@ -206,3 +207,26 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): # FIXME: DO NOT USE! Use django-send file or etc instead. return serve(request, basename, dirname) + + +class ResetViewSet(BaseViewSet): + allowed_methods = ['POST'] + + def post(self, request, *args, **kwargs): + # Only run when in DEBUG mode! It's only used for tests + if not settings.DEBUG: + return HttpResponseBadRequest("Only allowed in debug mode.") + + # Only allow local users, for extra safety + if not getattr(request.user, User.USERNAME_FIELD).endswith('@localhost'): + return HttpResponseBadRequest("Endpoint not allowed for user.") + + # Delete all of the journal data for this user for a clear test env + request.user.collection_set.all().delete() + + # FIXME: also delete chunk files!!! + + return HttpResponse() + + +reset = ResetViewSet.as_view({'post': 'post'}) From cf06534d6d9ea70f9fa1183a5c6887364e20529e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 14 Apr 2020 18:29:56 +0300 Subject: [PATCH 094/511] Serializers: handle our variant of b64 (no padding, urlsafe). --- django_etesync/serializers.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 127d88c..37b119e 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -29,10 +29,11 @@ def generate_rev_uid(length=32): class BinaryBase64Field(serializers.Field): def to_representation(self, value): - return base64.b64encode(value).decode('ascii') + return base64.urlsafe_b64encode(value).decode('ascii') def to_internal_value(self, data): - return base64.b64decode(data) + data += "=" * ((4 - len(data) % 4) % 4) + return base64.urlsafe_b64decode(data) class CollectionEncryptionKeyField(BinaryBase64Field): From a97bb969e78114982f0bb82010148fcf9b71312b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 14 Apr 2020 18:30:07 +0300 Subject: [PATCH 095/511] Make meta a binary base64 field too. --- django_etesync/serializers.py | 1 + 1 file changed, 1 insertion(+) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 37b119e..f0bb1c3 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -64,6 +64,7 @@ class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): queryset=models.CollectionItemChunk.objects.all(), many=True ) + meta = BinaryBase64Field() class Meta: model = models.CollectionItemRevision From 5dfa2ac8cbab2f0aa62158ff0f80e55726f7c898 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 14:33:38 +0300 Subject: [PATCH 096/511] Make chunks use the same b64 encoding we use elsewhere. --- django_etesync/serializers.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index f0bb1c3..2310c8d 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -27,13 +27,21 @@ def generate_rev_uid(length=32): return get_random_string(length) +def b64encode(value): + return base64.urlsafe_b64encode(value).decode('ascii') + + +def b64decode(data): + data += "=" * ((4 - len(data) % 4) % 4) + return base64.urlsafe_b64decode(data) + + class BinaryBase64Field(serializers.Field): def to_representation(self, value): - return base64.urlsafe_b64encode(value).decode('ascii') + return b64encode(value) def to_internal_value(self, data): - data += "=" * ((4 - len(data) % 4) % 4) - return base64.urlsafe_b64decode(data) + return b64decode(data) class CollectionEncryptionKeyField(BinaryBase64Field): @@ -99,7 +107,7 @@ class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerialize ret = [] for chunk in obj.chunks.all(): with open(chunk.chunkFile.path, 'rb') as f: - ret.append(base64.b64encode(f.read()).decode('ascii')) + ret.append(b64encode(f.read())) return ret From 963dc3c62df8457a64492b15aeba03b71e97009e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 15:23:07 +0300 Subject: [PATCH 097/511] Cleanup how we handle inline serializers. --- django_etesync/serializers.py | 42 ++++++++++++++++++----------------- django_etesync/views.py | 23 +++++++++++-------- 2 files changed, 36 insertions(+), 29 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 2310c8d..6a495bf 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -66,44 +66,40 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): fields = ('uid', 'chunkFile') -class CollectionItemRevisionBaseSerializer(serializers.ModelSerializer): +class CollectionItemRevisionSerializer(serializers.ModelSerializer): chunks = serializers.SlugRelatedField( slug_field='uid', queryset=models.CollectionItemChunk.objects.all(), many=True ) + chunksUrls = serializers.SerializerMethodField('get_chunks_urls') + chunksData = serializers.SerializerMethodField('get_chunks_data') meta = BinaryBase64Field() class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'meta', 'uid', 'deleted') - - -class CollectionItemRevisionSerializer(CollectionItemRevisionBaseSerializer): - chunksUrls = serializers.SerializerMethodField('get_chunks_urls') - - class Meta(CollectionItemRevisionBaseSerializer.Meta): - fields = CollectionItemRevisionBaseSerializer.Meta.fields + ('chunksUrls', ) + fields = ('chunks', 'meta', 'uid', 'deleted', 'chunksUrls', 'chunksData') # FIXME: currently the user is exposed in the url. We don't want that, and we can probably avoid that but still # save it under the user. # We would probably be better off just let the user calculate the urls from the uid and a base url for the snapshot. # E.g. chunkBaseUrl: "/media/bla/bla/" or chunkBaseUrl: "https://media.etesync.com/bla/bla" def get_chunks_urls(self, obj): + prefer_inline = self.context.get('prefer_inline', False) + if prefer_inline: + return None + ret = [] for chunk in obj.chunks.all(): ret.append(chunk.chunkFile.url) return ret - -class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerializer): - chunksData = serializers.SerializerMethodField('get_chunks_data') - - class Meta(CollectionItemRevisionBaseSerializer.Meta): - fields = CollectionItemRevisionBaseSerializer.Meta.fields + ('chunksData', ) - def get_chunks_data(self, obj): + prefer_inline = self.context.get('prefer_inline', False) + if not prefer_inline: + return None + ret = [] for chunk in obj.chunks.all(): with open(chunk.chunkFile.path, 'rb') as f: @@ -111,6 +107,16 @@ class CollectionItemRevisionInlineSerializer(CollectionItemRevisionBaseSerialize return ret + def to_representation(self, instance): + ret = super().to_representation(instance) + prefer_inline = self.context.get('prefer_inline', False) + if prefer_inline: + ret.pop('chunksUrls') + else: + ret.pop('chunksData') + + return ret + class CollectionItemSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() @@ -154,10 +160,6 @@ class CollectionItemSerializer(serializers.ModelSerializer): return instance -class CollectionItemInlineSerializer(CollectionItemSerializer): - content = CollectionItemRevisionInlineSerializer(read_only=True, many=False) - - class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') diff --git a/django_etesync/views.py b/django_etesync/views.py index a4b8108..37de4bb 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -29,7 +29,6 @@ from .models import Collection, CollectionItem from .serializers import ( CollectionSerializer, CollectionItemSerializer, - CollectionItemInlineSerializer, CollectionItemRevisionSerializer, CollectionItemChunkSerializer ) @@ -66,12 +65,18 @@ class CollectionViewSet(BaseViewSet): queryset = type(self).queryset return self.get_collection_queryset(queryset) + def get_serializer_context(self): + context = super().get_serializer_context() + prefer_inline = self.request.method == 'GET' and 'prefer_inline' in self.request.query_params + context.update({'request': self.request, 'prefer_inline': prefer_inline}) + return context + def destroy(self, request, uid=None): # FIXME: implement return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def create(self, request, *args, **kwargs): - serializer = self.serializer_class(data=request.data) + serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) if serializer.is_valid(): try: serializer.save(owner=self.request.user) @@ -86,7 +91,7 @@ class CollectionViewSet(BaseViewSet): def list(self, request): queryset = self.get_queryset() - serializer = self.serializer_class(queryset, context={'request': request}, many=True) + serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) return Response(serializer.data) @@ -98,12 +103,6 @@ class CollectionItemViewSet(BaseViewSet): pagination_class = paginators.LinkHeaderPagination lookup_field = 'uid' - def get_serializer_class(self): - if self.request.method == 'GET' and self.request.query_params.get('prefer_inline'): - return CollectionItemInlineSerializer - - return super().get_serializer_class() - def get_queryset(self): collection_uid = self.kwargs['collection_uid'] try: @@ -117,6 +116,12 @@ class CollectionItemViewSet(BaseViewSet): return queryset + def get_serializer_context(self): + context = super().get_serializer_context() + prefer_inline = self.request.method == 'GET' and 'prefer_inline' in self.request.query_params + context.update({'request': self.request, 'prefer_inline': prefer_inline}) + return context + def create(self, request, collection_uid=None): collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) From c589d06cbe1283e42d6c951530d547c32ce1e730 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:00:06 +0300 Subject: [PATCH 098/511] Collection: lax the restrictions on mainItem. --- .../migrations/0034_auto_20200415_1248.py | 19 +++++++++++++++++++ .../migrations/0035_auto_20200415_1259.py | 19 +++++++++++++++++++ django_etesync/models.py | 2 +- 3 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0034_auto_20200415_1248.py create mode 100644 django_etesync/migrations/0035_auto_20200415_1259.py diff --git a/django_etesync/migrations/0034_auto_20200415_1248.py b/django_etesync/migrations/0034_auto_20200415_1248.py new file mode 100644 index 0000000..1156676 --- /dev/null +++ b/django_etesync/migrations/0034_auto_20200415_1248.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-04-15 12:48 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0033_auto_20200317_2010'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='mainItem', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, related_name='of_collection', to='django_etesync.CollectionItem'), + ), + ] diff --git a/django_etesync/migrations/0035_auto_20200415_1259.py b/django_etesync/migrations/0035_auto_20200415_1259.py new file mode 100644 index 0000000..d558e31 --- /dev/null +++ b/django_etesync/migrations/0035_auto_20200415_1259.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-04-15 12:59 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0034_auto_20200415_1248'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='mainItem', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='of_collection', to='django_etesync.CollectionItem'), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index ca3dea9..65f130a 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -29,7 +29,7 @@ class Collection(models.Model): max_length=44, validators=[UidValidator]) version = models.PositiveSmallIntegerField() owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) - mainItem = models.OneToOneField('CollectionItem', related_name='of_collection', on_delete=models.PROTECT) + mainItem = models.OneToOneField('CollectionItem', related_name='of_collection', null=True, on_delete=models.SET_NULL) class Meta: unique_together = ('uid', 'owner') From 6711cfcf493a0e5dc65ae96703b4ea5ac59266fc Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:27:03 +0300 Subject: [PATCH 099/511] Change chunks to be just one field. --- django_etesync/serializers.py | 57 +++++++++-------------------------- 1 file changed, 15 insertions(+), 42 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 6a495bf..5946293 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -60,6 +60,19 @@ class CollectionContentField(BinaryBase64Field): return None +class ChunksField(serializers.RelatedField): + def to_representation(self, obj): + prefer_inline = self.context.get('prefer_inline', False) + if prefer_inline: + with open(obj.chunkFile.path, 'rb') as f: + return (obj.uid, b64encode(f.read())) + else: + return (obj.uid, ) + + def to_internal_value(self, data): + return (data[0], b64decode(data[1])) + + class CollectionItemChunkSerializer(serializers.ModelSerializer): class Meta: model = models.CollectionItemChunk @@ -67,55 +80,15 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): class CollectionItemRevisionSerializer(serializers.ModelSerializer): - chunks = serializers.SlugRelatedField( - slug_field='uid', + chunks = ChunksField( queryset=models.CollectionItemChunk.objects.all(), many=True ) - chunksUrls = serializers.SerializerMethodField('get_chunks_urls') - chunksData = serializers.SerializerMethodField('get_chunks_data') meta = BinaryBase64Field() class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'meta', 'uid', 'deleted', 'chunksUrls', 'chunksData') - - # FIXME: currently the user is exposed in the url. We don't want that, and we can probably avoid that but still - # save it under the user. - # We would probably be better off just let the user calculate the urls from the uid and a base url for the snapshot. - # E.g. chunkBaseUrl: "/media/bla/bla/" or chunkBaseUrl: "https://media.etesync.com/bla/bla" - def get_chunks_urls(self, obj): - prefer_inline = self.context.get('prefer_inline', False) - if prefer_inline: - return None - - ret = [] - for chunk in obj.chunks.all(): - ret.append(chunk.chunkFile.url) - - return ret - - def get_chunks_data(self, obj): - prefer_inline = self.context.get('prefer_inline', False) - if not prefer_inline: - return None - - ret = [] - for chunk in obj.chunks.all(): - with open(chunk.chunkFile.path, 'rb') as f: - ret.append(b64encode(f.read())) - - return ret - - def to_representation(self, instance): - ret = super().to_representation(instance) - prefer_inline = self.context.get('prefer_inline', False) - if prefer_inline: - ret.pop('chunksUrls') - else: - ret.pop('chunksData') - - return ret + fields = ('chunks', 'meta', 'uid', 'deleted') class CollectionItemSerializer(serializers.ModelSerializer): From 2e018dfe7672e6fd4813c410f0f0e386b392ead8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:47:31 +0300 Subject: [PATCH 100/511] Rename prefer_inline to inline. --- django_etesync/serializers.py | 4 ++-- django_etesync/views.py | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 5946293..bc04bc0 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -62,8 +62,8 @@ class CollectionContentField(BinaryBase64Field): class ChunksField(serializers.RelatedField): def to_representation(self, obj): - prefer_inline = self.context.get('prefer_inline', False) - if prefer_inline: + inline = self.context.get('inline', False) + if inline: with open(obj.chunkFile.path, 'rb') as f: return (obj.uid, b64encode(f.read())) else: diff --git a/django_etesync/views.py b/django_etesync/views.py index 37de4bb..1c9da78 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -67,8 +67,8 @@ class CollectionViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefer_inline = self.request.method == 'GET' and 'prefer_inline' in self.request.query_params - context.update({'request': self.request, 'prefer_inline': prefer_inline}) + inline = self.request.method == 'GET' and 'inline' in self.request.query_params + context.update({'request': self.request, 'inline': inline}) return context def destroy(self, request, uid=None): @@ -118,8 +118,8 @@ class CollectionItemViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefer_inline = self.request.method == 'GET' and 'prefer_inline' in self.request.query_params - context.update({'request': self.request, 'prefer_inline': prefer_inline}) + inline = self.request.method == 'GET' and 'inline' in self.request.query_params + context.update({'request': self.request, 'inline': inline}) return context def create(self, request, collection_uid=None): From 3db204e4bb52163e3bc2ddd2063fd858fb9ec3b0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:50:47 +0300 Subject: [PATCH 101/511] b64: don't add redundant padding. --- django_etesync/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index bc04bc0..1b87a86 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -28,7 +28,7 @@ def generate_rev_uid(length=32): def b64encode(value): - return base64.urlsafe_b64encode(value).decode('ascii') + return base64.urlsafe_b64encode(value).decode('ascii').strip('=') def b64decode(data): From 6dfa2360c090e6f0768b63146d108dae91d23293 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:52:36 +0300 Subject: [PATCH 102/511] Chunk: fix for a collection's main item. --- django_etesync/models.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 65f130a..fdf8041 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -64,7 +64,8 @@ def chunk_directory_path(instance, filename): item = instance.item col = item.collection user_id = col.owner.id - return Path('user_{}'.format(user_id), col.uid, item.uid, instance.uid) + item_uid = item.uid or 'main' + return Path('user_{}'.format(user_id), col.uid, item_uid, instance.uid) class CollectionItemChunk(models.Model): From ab9d66fcc0da829ce3bed28d1d436ea48206a168 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:53:31 +0300 Subject: [PATCH 103/511] Implement collection creation. --- django_etesync/serializers.py | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 1b87a86..4d8a206 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -14,6 +14,7 @@ import base64 +from django.core.files.base import ContentFile from django.contrib.auth import get_user_model from django.db import transaction from django.utils.crypto import get_random_string @@ -164,14 +165,28 @@ class CollectionSerializer(serializers.ModelSerializer): instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): + instance.save() main_item = models.CollectionItem.objects.create( uid=None, encryptionKey=None, version=instance.version, collection=instance) instance.mainItem = main_item + chunks_ids = [] chunks = revision_data.pop('chunks') - revision = models.CollectionItemRevision.objects.create(**revision_data, uid=generate_rev_uid(), - item=main_item) - revision.chunks.set(chunks) + for chunk in chunks: + uid = chunk[0] + if len(chunk) > 1: + content = chunk[1] + # FIXME: fix order! + chunk = models.CollectionItemChunk(uid=uid, item=main_item, order='abc') + chunk.chunkFile.save('IGNORED', ContentFile(content)) + chunk.save() + chunks_ids.append(chunk.id) + else: + chunk = models.CollectionItemChunk.objects.get(uid=uid) + chunks_ids.append(chunk.id) + + revision = models.CollectionItemRevision.objects.create(**revision_data, item=main_item) + revision.chunks.set(chunks_ids) instance.save() models.CollectionMember(collection=instance, From 7a0a00c738f9795324e7610102fef468a0e63ef8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 16:59:30 +0300 Subject: [PATCH 104/511] Unify how we handle revision creation. --- django_etesync/serializers.py | 49 +++++++++++++++++------------------ 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 4d8a206..60c7560 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -28,6 +28,27 @@ def generate_rev_uid(length=32): return get_random_string(length) +def process_revisions_for_item(item, revision_data): + chunks_ids = [] + chunks = revision_data.pop('chunks') + for chunk in chunks: + uid = chunk[0] + if len(chunk) > 1: + content = chunk[1] + # FIXME: fix order! + chunk = models.CollectionItemChunk(uid=uid, item=item, order='abc') + chunk.chunkFile.save('IGNORED', ContentFile(content)) + chunk.save() + chunks_ids.append(chunk.id) + else: + chunk = models.CollectionItemChunk.objects.get(uid=uid) + chunks_ids.append(chunk.id) + + revision = models.CollectionItemRevision.objects.create(**revision_data, item=item) + revision.chunks.set(chunks_ids) + return revision + + def b64encode(value): return base64.urlsafe_b64encode(value).decode('ascii').strip('=') @@ -108,10 +129,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): with transaction.atomic(): instance.save() - chunks = revision_data.pop('chunks') - revision = models.CollectionItemRevision.objects.create(**revision_data, uid=generate_rev_uid(), - item=instance) - revision.chunks.set(chunks) + process_revisions_for_item(instance, revision_data) return instance @@ -126,10 +144,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): current_revision.current = None current_revision.save() - chunks = revision_data.pop('chunks') - revision = models.CollectionItemRevision.objects.create(**revision_data, uid=generate_rev_uid(), - item=instance) - revision.chunks.set(chunks) + process_revisions_for_item(instance, revision_data) return instance @@ -170,23 +185,7 @@ class CollectionSerializer(serializers.ModelSerializer): uid=None, encryptionKey=None, version=instance.version, collection=instance) instance.mainItem = main_item - chunks_ids = [] - chunks = revision_data.pop('chunks') - for chunk in chunks: - uid = chunk[0] - if len(chunk) > 1: - content = chunk[1] - # FIXME: fix order! - chunk = models.CollectionItemChunk(uid=uid, item=main_item, order='abc') - chunk.chunkFile.save('IGNORED', ContentFile(content)) - chunk.save() - chunks_ids.append(chunk.id) - else: - chunk = models.CollectionItemChunk.objects.get(uid=uid) - chunks_ids.append(chunk.id) - - revision = models.CollectionItemRevision.objects.create(**revision_data, item=main_item) - revision.chunks.set(chunks_ids) + process_revisions_for_item(main_item, revision_data) instance.save() models.CollectionMember(collection=instance, From 62a7496b66b83284fd0dad4d59042149c99720bf Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 17:35:51 +0300 Subject: [PATCH 105/511] Change how we handle chunk ordering (and relation). --- .../migrations/0036_auto_20200415_1420.py | 37 +++++++++++++++++++ .../migrations/0037_auto_20200415_1421.py | 23 ++++++++++++ ...38_remove_collectionitemrevision_chunks.py | 17 +++++++++ django_etesync/models.py | 14 ++++--- django_etesync/serializers.py | 18 +++++---- django_etesync/views.py | 4 +- 6 files changed, 96 insertions(+), 17 deletions(-) create mode 100644 django_etesync/migrations/0036_auto_20200415_1420.py create mode 100644 django_etesync/migrations/0037_auto_20200415_1421.py create mode 100644 django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py diff --git a/django_etesync/migrations/0036_auto_20200415_1420.py b/django_etesync/migrations/0036_auto_20200415_1420.py new file mode 100644 index 0000000..a7b8003 --- /dev/null +++ b/django_etesync/migrations/0036_auto_20200415_1420.py @@ -0,0 +1,37 @@ +# Generated by Django 3.0.3 on 2020-04-15 14:20 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0035_auto_20200415_1259'), + ] + + operations = [ + migrations.AlterModelOptions( + name='collectionitemchunk', + options={}, + ), + migrations.AlterUniqueTogether( + name='collectionitemchunk', + unique_together=set(), + ), + migrations.CreateModel( + name='RevisionChunkRelation', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('chunk', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions_relation', to='django_etesync.CollectionItemChunk')), + ('revision', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks_relation', to='django_etesync.CollectionItemRevision')), + ], + options={ + 'ordering': ('id',), + }, + ), + migrations.RemoveField( + model_name='collectionitemchunk', + name='order', + ), + ] diff --git a/django_etesync/migrations/0037_auto_20200415_1421.py b/django_etesync/migrations/0037_auto_20200415_1421.py new file mode 100644 index 0000000..d1a47db --- /dev/null +++ b/django_etesync/migrations/0037_auto_20200415_1421.py @@ -0,0 +1,23 @@ +# Generated by Django 3.0.3 on 2020-04-15 14:21 + +from django.db import migrations + + +def change_chunk_relation(apps, schema_editor): + CollectionItemRevision = apps.get_model('django_etesync', 'CollectionItemRevision') + RevisionChunkRelation = apps.get_model('django_etesync', 'RevisionChunkRelation') + + for revision in CollectionItemRevision.objects.all(): + for chunk in revision.chunks.all(): + RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0036_auto_20200415_1420'), + ] + + operations = [ + migrations.RunPython(change_chunk_relation), + ] diff --git a/django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py b/django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py new file mode 100644 index 0000000..6e35b86 --- /dev/null +++ b/django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py @@ -0,0 +1,17 @@ +# Generated by Django 3.0.3 on 2020-04-15 14:34 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0037_auto_20200415_1421'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitemrevision', + name='chunks', + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index fdf8041..449c743 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -72,13 +72,8 @@ class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=44, validators=[Base64Url256BitValidator]) item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) - order = models.CharField(max_length=100, blank=False, null=False) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) - class Meta: - unique_together = ('item', 'order') - ordering = ('item', 'order') - def __str__(self): return self.uid @@ -88,7 +83,6 @@ class CollectionItemRevision(models.Model): max_length=44, validators=[Base64Url256BitValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=False, null=False) - chunks = models.ManyToManyField(CollectionItemChunk, related_name='items') current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) @@ -99,6 +93,14 @@ class CollectionItemRevision(models.Model): return '{} {} current={}'.format(self.uid, self.item.uid, self.current) +class RevisionChunkRelation(models.Model): + chunk = models.ForeignKey(CollectionItemChunk, related_name='revisions_relation', on_delete=models.CASCADE) + revision = models.ForeignKey(CollectionItemRevision, related_name='chunks_relation', on_delete=models.CASCADE) + + class Meta: + ordering = ('id', ) + + class CollectionMember(models.Model): class AccessLevels(models.TextChoices): ADMIN = 'adm' diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 60c7560..f7f4b71 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -29,23 +29,23 @@ def generate_rev_uid(length=32): def process_revisions_for_item(item, revision_data): - chunks_ids = [] - chunks = revision_data.pop('chunks') + chunks_objs = [] + chunks = revision_data.pop('chunks_relation') for chunk in chunks: uid = chunk[0] if len(chunk) > 1: content = chunk[1] - # FIXME: fix order! - chunk = models.CollectionItemChunk(uid=uid, item=item, order='abc') + chunk = models.CollectionItemChunk(uid=uid, item=item) chunk.chunkFile.save('IGNORED', ContentFile(content)) chunk.save() - chunks_ids.append(chunk.id) + chunks_objs.append(chunk) else: chunk = models.CollectionItemChunk.objects.get(uid=uid) - chunks_ids.append(chunk.id) + chunks_objs.append(chunk) revision = models.CollectionItemRevision.objects.create(**revision_data, item=item) - revision.chunks.set(chunks_ids) + for chunk in chunks_objs: + models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) return revision @@ -84,6 +84,7 @@ class CollectionContentField(BinaryBase64Field): class ChunksField(serializers.RelatedField): def to_representation(self, obj): + obj = obj.chunk inline = self.context.get('inline', False) if inline: with open(obj.chunkFile.path, 'rb') as f: @@ -103,7 +104,8 @@ class CollectionItemChunkSerializer(serializers.ModelSerializer): class CollectionItemRevisionSerializer(serializers.ModelSerializer): chunks = ChunksField( - queryset=models.CollectionItemChunk.objects.all(), + source='chunks_relation', + queryset=models.RevisionChunkRelation.objects.all(), many=True ) meta = BinaryBase64Field() diff --git a/django_etesync/views.py b/django_etesync/views.py index 1c9da78..c4c8479 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -186,9 +186,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer = self.serializer_class(data=request.data) if serializer.is_valid(): try: - # FIXME: actually generate the correct order value. Or alternatively have it null at first and only - # set it when ommitting to a snapshot - serializer.save(item=col_it, order='abc') + serializer.save(item=col_it) except IntegrityError: content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) From 0fbc5c104c0359be3c37747b10fadbdd167f8591 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 15 Apr 2020 17:54:39 +0300 Subject: [PATCH 106/511] Implement collection updating. --- django_etesync/serializers.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index f7f4b71..d4b809f 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -197,3 +197,19 @@ class CollectionSerializer(serializers.ModelSerializer): ).save() return instance + + def update(self, instance, validated_data): + """Function that's called when this serializer is meant to update an item""" + revision_data = validated_data.pop('content') + + with transaction.atomic(): + main_item = instance.mainItem + # We don't have to use select_for_update here because the unique constraint on current guards against + # the race condition. But it's a good idea because it'll lock and wait rather than fail. + current_revision = main_item.revisions.filter(current=True).select_for_update().first() + current_revision.current = None + current_revision.save() + + process_revisions_for_item(main_item, revision_data) + + return instance From a72543f6c9ae10f8f9bff8bd439dbbef7159703d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 11:28:49 +0300 Subject: [PATCH 107/511] Collection remove the redundant mainItem model attr. --- .../0039_remove_collection_mainitem.py | 17 +++++++++++++++++ django_etesync/models.py | 7 +++++-- django_etesync/serializers.py | 4 +--- 3 files changed, 23 insertions(+), 5 deletions(-) create mode 100644 django_etesync/migrations/0039_remove_collection_mainitem.py diff --git a/django_etesync/migrations/0039_remove_collection_mainitem.py b/django_etesync/migrations/0039_remove_collection_mainitem.py new file mode 100644 index 0000000..1822bc7 --- /dev/null +++ b/django_etesync/migrations/0039_remove_collection_mainitem.py @@ -0,0 +1,17 @@ +# Generated by Django 3.0.3 on 2020-04-16 08:28 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0038_remove_collectionitemrevision_chunks'), + ] + + operations = [ + migrations.RemoveField( + model_name='collection', + name='mainItem', + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 449c743..7315ee8 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -29,7 +29,6 @@ class Collection(models.Model): max_length=44, validators=[UidValidator]) version = models.PositiveSmallIntegerField() owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) - mainItem = models.OneToOneField('CollectionItem', related_name='of_collection', null=True, on_delete=models.SET_NULL) class Meta: unique_together = ('uid', 'owner') @@ -37,9 +36,13 @@ class Collection(models.Model): def __str__(self): return self.uid + @cached_property + def main_item(self): + return self.items.get(uid=None) + @cached_property def content(self): - return self.mainItem.content + return self.main_item.content class CollectionItem(models.Model): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index d4b809f..b2d6683 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -185,11 +185,9 @@ class CollectionSerializer(serializers.ModelSerializer): instance.save() main_item = models.CollectionItem.objects.create( uid=None, encryptionKey=None, version=instance.version, collection=instance) - instance.mainItem = main_item process_revisions_for_item(main_item, revision_data) - instance.save() models.CollectionMember(collection=instance, user=validated_data.get('owner'), accessLevel=models.CollectionMember.AccessLevels.ADMIN, @@ -203,7 +201,7 @@ class CollectionSerializer(serializers.ModelSerializer): revision_data = validated_data.pop('content') with transaction.atomic(): - main_item = instance.mainItem + main_item = instance.main_item # We don't have to use select_for_update here because the unique constraint on current guards against # the race condition. But it's a good idea because it'll lock and wait rather than fail. current_revision = main_item.revisions.filter(current=True).select_for_update().first() From ca7d7dfd1287447f3215cf78e05277ae3b373882 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 11:35:58 +0300 Subject: [PATCH 108/511] Allow passing inline to not only GET requests. --- django_etesync/views.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index c4c8479..6c2faf3 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -67,7 +67,7 @@ class CollectionViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - inline = self.request.method == 'GET' and 'inline' in self.request.query_params + inline = 'inline' in self.request.query_params context.update({'request': self.request, 'inline': inline}) return context @@ -118,7 +118,7 @@ class CollectionItemViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - inline = self.request.method == 'GET' and 'inline' in self.request.query_params + inline = 'inline' in self.request.query_params context.update({'request': self.request, 'inline': inline}) return context From 1f97d1dbf7ccb50c6b693042d5958055751e51b3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 12:56:42 +0300 Subject: [PATCH 109/511] Remove redundant gen_rev_uid. --- django_etesync/migrations/0029_auto_20200312_1849.py | 6 +++++- django_etesync/serializers.py | 5 ----- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/django_etesync/migrations/0029_auto_20200312_1849.py b/django_etesync/migrations/0029_auto_20200312_1849.py index 165b405..568aaa2 100644 --- a/django_etesync/migrations/0029_auto_20200312_1849.py +++ b/django_etesync/migrations/0029_auto_20200312_1849.py @@ -1,7 +1,11 @@ # Generated by Django 3.0.3 on 2020-03-12 18:49 from django.db import migrations -from django_etesync.serializers import generate_rev_uid +from django.utils.crypto import get_random_string + + +def generate_rev_uid(length=32): + return get_random_string(length) def add_collection_main_item(apps, schema_editor): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index b2d6683..4d1bfa5 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -17,17 +17,12 @@ import base64 from django.core.files.base import ContentFile from django.contrib.auth import get_user_model from django.db import transaction -from django.utils.crypto import get_random_string from rest_framework import serializers from . import models User = get_user_model() -def generate_rev_uid(length=32): - return get_random_string(length) - - def process_revisions_for_item(item, revision_data): chunks_objs = [] chunks = revision_data.pop('chunks_relation') From edaa7b0f0533c7658ffadd91cd28dc02a83eb44e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 15:35:44 +0300 Subject: [PATCH 110/511] Rename ctag to stoken. --- django_etesync/serializers.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 4d1bfa5..03dd0bb 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -149,12 +149,12 @@ class CollectionItemSerializer(serializers.ModelSerializer): class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') - ctag = serializers.SerializerMethodField('get_ctag') + stoken = serializers.SerializerMethodField('get_stoken') content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.Collection - fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'ctag') + fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'stoken') def get_access_level_from_context(self, obj): request = self.context.get('request', None) @@ -162,7 +162,7 @@ class CollectionSerializer(serializers.ModelSerializer): return obj.members.get(user=request.user).accessLevel return None - def get_ctag(self, obj): + def get_stoken(self, obj): last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() if last_revision is None: # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. From c5af5fd4e6b2ee26b9658c9c95a9b9b5294d2d48 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 16:33:16 +0300 Subject: [PATCH 111/511] Collection: move stoken to the model. --- django_etesync/models.py | 9 +++++++++ django_etesync/serializers.py | 10 +--------- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 7315ee8..c6ec571 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -44,6 +44,15 @@ class Collection(models.Model): def content(self): return self.main_item.content + @cached_property + def stoken(self): + last_revision = CollectionItemRevision.objects.filter(item__collection=self).last() + if last_revision is None: + # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. + return None + + return last_revision.uid + class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=True, diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 03dd0bb..eb50d76 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -149,7 +149,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') - stoken = serializers.SerializerMethodField('get_stoken') + stoken = serializers.CharField(read_only=True) content = CollectionItemRevisionSerializer(many=False) class Meta: @@ -162,14 +162,6 @@ class CollectionSerializer(serializers.ModelSerializer): return obj.members.get(user=request.user).accessLevel return None - def get_stoken(self, obj): - last_revision = models.CollectionItemRevision.objects.filter(item__collection=obj).last() - if last_revision is None: - # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. - return None - - return last_revision.uid - def create(self, validated_data): """Function that's called when this serializer creates an item""" revision_data = validated_data.pop('content') From 687bf9924b80f10e852d69eaa78c291c3dd99763 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 16:37:26 +0300 Subject: [PATCH 112/511] API: change how pagination and stoken are done --- django_etesync/paginators.py | 36 ---------------------------- django_etesync/views.py | 46 +++++++++++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 42 deletions(-) delete mode 100644 django_etesync/paginators.py diff --git a/django_etesync/paginators.py b/django_etesync/paginators.py deleted file mode 100644 index 6d55599..0000000 --- a/django_etesync/paginators.py +++ /dev/null @@ -1,36 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -from rest_framework import pagination -from rest_framework.response import Response - - -class LinkHeaderPagination(pagination.LimitOffsetPagination): - def get_paginated_response(self, data): - next_url = self.get_next_link() - previous_url = self.get_previous_link() - - if next_url is not None and previous_url is not None: - link = '<{next_url}>; rel="next", <{previous_url}>; rel="prev"' - elif next_url is not None: - link = '<{next_url}>; rel="next"' - elif previous_url is not None: - link = '<{previous_url}>; rel="prev"' - else: - link = '' - - link = link.format(next_url=next_url, previous_url=previous_url) - headers = {'Link': link} if link else {} - - return Response(data, headers=headers) diff --git a/django_etesync/views.py b/django_etesync/views.py index 6c2faf3..df22068 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -24,8 +24,8 @@ from rest_framework import parsers from rest_framework.decorators import action as action_decorator from rest_framework.response import Response -from . import app_settings, paginators -from .models import Collection, CollectionItem +from . import app_settings +from .models import Collection, CollectionItem, CollectionItemRevision from .serializers import ( CollectionSerializer, CollectionItemSerializer, @@ -61,8 +61,9 @@ class CollectionViewSet(BaseViewSet): serializer_class = CollectionSerializer lookup_field = 'uid' - def get_queryset(self): - queryset = type(self).queryset + def get_queryset(self, queryset=None): + if queryset is None: + queryset = type(self).queryset return self.get_collection_queryset(queryset) def get_serializer_context(self): @@ -89,10 +90,24 @@ class CollectionViewSet(BaseViewSet): return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) def list(self, request): + stoken = request.GET.get('stoken', None) + limit = int(request.GET.get('limit', 50)) + queryset = self.get_queryset() + if stoken is not None: + last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + queryset = queryset.filter(items__revisions__id__gt=last_rev.id) + + queryset = queryset[:limit] + serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) - return Response(serializer.data) + + new_stoken = serializer.data[-1]['stoken'] if len(serializer.data) > 0 else stoken + ret = { + 'data': serializer.data, + } + return Response(ret, headers={'X-EteSync-SToken': new_stoken}) class CollectionItemViewSet(BaseViewSet): @@ -100,7 +115,6 @@ class CollectionItemViewSet(BaseViewSet): permission_classes = BaseViewSet.permission_classes queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer - pagination_class = paginators.LinkHeaderPagination lookup_field = 'uid' def get_queryset(self): @@ -148,6 +162,26 @@ class CollectionItemViewSet(BaseViewSet): # FIXME: implement, or should it be implemented elsewhere? return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + def list(self, request, collection_uid=None): + stoken = request.GET.get('stoken', None) + limit = int(request.GET.get('limit', 50)) + + queryset = self.get_queryset() + + if stoken is not None: + last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + queryset = queryset.filter(revisions__id__gt=last_rev.id) + + queryset = queryset[:limit] + + serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) + + new_stoken = serializer.data[-1]['content']['uid'] if len(serializer.data) > 0 else stoken + ret = { + 'data': serializer.data, + } + return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) From 19b93265d788e9a2c4f5661de7d15a13e21c9c93 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 16:43:21 +0300 Subject: [PATCH 113/511] Add a comment. --- django_etesync/views.py | 1 + 1 file changed, 1 insertion(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index df22068..e8bd6de 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -184,6 +184,7 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): + # FIXME: need pagination support col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) col_it = get_object_or_404(col.items, uid=uid) From d134934f8c190f9b11129ae847c8857906369da2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 17:12:51 +0300 Subject: [PATCH 114/511] Bulk_get: implement stoken, limit and inline --- django_etesync/views.py | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index e8bd6de..d9b6283 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -164,18 +164,14 @@ class CollectionItemViewSet(BaseViewSet): def list(self, request, collection_uid=None): stoken = request.GET.get('stoken', None) - limit = int(request.GET.get('limit', 50)) queryset = self.get_queryset() - - if stoken is not None: - last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) - queryset = queryset.filter(revisions__id__gt=last_rev.id) - - queryset = queryset[:limit] + queryset = self.filter_by_stoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) + queryset = self.filter_by_stoken_and_limit(request, queryset) + new_stoken = serializer.data[-1]['content']['uid'] if len(serializer.data) > 0 else stoken ret = { 'data': serializer.data, @@ -193,13 +189,31 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def bulk_get(self, request, collection_uid=None): + stoken = request.GET.get('stoken', None) queryset = self.get_queryset() if isinstance(request.data, list): queryset = queryset.filter(uid__in=request.data) - serializer = self.get_serializer_class()(queryset, many=True) - return Response(serializer.data, status=status.HTTP_200_OK) + queryset = self.filter_by_stoken_and_limit(request, queryset) + + serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) + + new_stoken = serializer.data[-1]['content']['uid'] if len(serializer.data) > 0 else stoken + ret = { + 'data': serializer.data, + } + return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + + def filter_by_stoken_and_limit(self, request, queryset): + stoken = request.GET.get('stoken', None) + limit = int(request.GET.get('limit', 50)) + + if stoken is not None: + last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + queryset = queryset.filter(revisions__id__gt=last_rev.id) + + return queryset[:limit] class CollectionItemChunkViewSet(viewsets.ViewSet): From f23815d46d348a95401aeeeca485d105b68a7c01 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 17:14:03 +0300 Subject: [PATCH 115/511] Fix calculation of stoken. --- django_etesync/views.py | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index d9b6283..53e6adb 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -15,6 +15,7 @@ from django.conf import settings from django.contrib.auth import get_user_model from django.db import IntegrityError +from django.db.models import Max from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -103,7 +104,9 @@ class CollectionViewSet(BaseViewSet): serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) - new_stoken = serializer.data[-1]['stoken'] if len(serializer.data) > 0 else stoken + new_stoken_id = queryset.aggregate(stoken_id=Max('items__revisions__id'))['stoken_id'] + new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken + ret = { 'data': serializer.data, } @@ -163,16 +166,11 @@ class CollectionItemViewSet(BaseViewSet): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def list(self, request, collection_uid=None): - stoken = request.GET.get('stoken', None) - queryset = self.get_queryset() - queryset = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) - queryset = self.filter_by_stoken_and_limit(request, queryset) - - new_stoken = serializer.data[-1]['content']['uid'] if len(serializer.data) > 0 else stoken ret = { 'data': serializer.data, } @@ -189,17 +187,15 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def bulk_get(self, request, collection_uid=None): - stoken = request.GET.get('stoken', None) queryset = self.get_queryset() if isinstance(request.data, list): queryset = queryset.filter(uid__in=request.data) - queryset = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) - new_stoken = serializer.data[-1]['content']['uid'] if len(serializer.data) > 0 else stoken ret = { 'data': serializer.data, } @@ -213,7 +209,10 @@ class CollectionItemViewSet(BaseViewSet): last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) queryset = queryset.filter(revisions__id__gt=last_rev.id) - return queryset[:limit] + new_stoken_id = queryset.aggregate(stoken_id=Max('revisions__id'))['stoken_id'] + new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken + + return queryset[:limit], new_stoken class CollectionItemChunkViewSet(viewsets.ViewSet): From 9f0f00a59496c57c84aa8cf553150f4fb6f92f37 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 17:36:06 +0300 Subject: [PATCH 116/511] Unify the stoken filtering and calculation. --- django_etesync/views.py | 45 ++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 53e6adb..25f2d2a 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -41,6 +41,7 @@ User = get_user_model() class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) + stoken_id_field = None def get_serializer_class(self): serializer_class = self.serializer_class @@ -54,6 +55,22 @@ class BaseViewSet(viewsets.ModelViewSet): user = self.request.user return queryset.filter(members__user=user) + def filter_by_stoken_and_limit(self, request, queryset): + stoken = request.GET.get('stoken', None) + limit = int(request.GET.get('limit', 50)) + + stoken_id_field = self.stoken_id_field + '__id' + + if stoken is not None: + last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + filter_by = {stoken_id_field + '__gt': last_rev.id} + queryset = queryset.filter(**filter_by) + + new_stoken_id = queryset.aggregate(stoken_id=Max(stoken_id_field))['stoken_id'] + new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken + + return queryset[:limit], new_stoken + class CollectionViewSet(BaseViewSet): allowed_methods = ['GET', 'POST', 'DELETE'] @@ -61,6 +78,7 @@ class CollectionViewSet(BaseViewSet): queryset = Collection.objects.all() serializer_class = CollectionSerializer lookup_field = 'uid' + stoken_id_field = 'items__revisions' def get_queryset(self, queryset=None): if queryset is None: @@ -91,22 +109,11 @@ class CollectionViewSet(BaseViewSet): return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) def list(self, request): - stoken = request.GET.get('stoken', None) - limit = int(request.GET.get('limit', 50)) - queryset = self.get_queryset() - - if stoken is not None: - last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) - queryset = queryset.filter(items__revisions__id__gt=last_rev.id) - - queryset = queryset[:limit] + queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) - new_stoken_id = queryset.aggregate(stoken_id=Max('items__revisions__id'))['stoken_id'] - new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken - ret = { 'data': serializer.data, } @@ -119,6 +126,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = 'uid' + stoken_id_field = 'revisions' def get_queryset(self): collection_uid = self.kwargs['collection_uid'] @@ -201,19 +209,6 @@ class CollectionItemViewSet(BaseViewSet): } return Response(ret, headers={'X-EteSync-SToken': new_stoken}) - def filter_by_stoken_and_limit(self, request, queryset): - stoken = request.GET.get('stoken', None) - limit = int(request.GET.get('limit', 50)) - - if stoken is not None: - last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) - queryset = queryset.filter(revisions__id__gt=last_rev.id) - - new_stoken_id = queryset.aggregate(stoken_id=Max('revisions__id'))['stoken_id'] - new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken - - return queryset[:limit], new_stoken - class CollectionItemChunkViewSet(viewsets.ViewSet): allowed_methods = ['GET', 'POST'] From af2787195554cfd624ab343b05f50004997709a2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Apr 2020 17:38:07 +0300 Subject: [PATCH 117/511] Revision: change the shape of the list response. --- django_etesync/views.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 25f2d2a..7d6234f 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -191,7 +191,10 @@ class CollectionItemViewSet(BaseViewSet): col_it = get_object_or_404(col.items, uid=uid) serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) - return Response(serializer.data) + ret = { + 'data': serializer.data, + } + return Response(ret) @action_decorator(detail=False, methods=['POST']) def bulk_get(self, request, collection_uid=None): From d66d0640dcfbebbcc6d375e356878e93e53503fc Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 19 Apr 2020 15:13:09 +0300 Subject: [PATCH 118/511] Collection: disallow partial updates. --- django_etesync/views.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 7d6234f..b9cd878 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -95,6 +95,9 @@ class CollectionViewSet(BaseViewSet): # FIXME: implement return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + def partial_update(self, request, uid=None): + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + def create(self, request, *args, **kwargs): serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) if serializer.is_valid(): From df0f7d134d5b07894762d07190b62019bf84a55e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 19 Apr 2020 17:32:40 +0300 Subject: [PATCH 119/511] Collection items: add a transaction endpoint. --- django_etesync/models.py | 4 ++++ django_etesync/views.py | 31 ++++++++++++++++++++++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index c6ec571..05b81b3 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -71,6 +71,10 @@ class CollectionItem(models.Model): def content(self): return self.revisions.get(current=True) + @cached_property + def stoken(self): + return self.content.uid + def chunk_directory_path(instance, filename): item = instance.item diff --git a/django_etesync/views.py b/django_etesync/views.py index b9cd878..87c8010 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -14,7 +14,7 @@ from django.conf import settings from django.contrib.auth import get_user_model -from django.db import IntegrityError +from django.db import transaction, IntegrityError from django.db.models import Max from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -215,6 +215,35 @@ class CollectionItemViewSet(BaseViewSet): } return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + @action_decorator(detail=False, methods=['POST']) + def transaction(self, request, collection_uid=None): + collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + + items = request.data.get('items') + # FIXME: deps should actually be just pairs of uid and stoken + deps = request.data.get('deps', None) + serializer = self.get_serializer_class()(data=items, context=self.get_serializer_context(), many=True) + deps_serializer = self.get_serializer_class()(data=deps, context=self.get_serializer_context(), many=True) + if serializer.is_valid() and (deps is None or deps_serializer.is_valid()): + try: + with transaction.atomic(): + collections = serializer.save(collection=collection_object) + except IntegrityError: + content = {'code': 'integrity_error'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + ret = { + "data": [collection.stoken for collection in collections], + } + return Response(ret, status=status.HTTP_201_CREATED) + + return Response( + { + "items": serializer.errors, + "deps": deps_serializer.errors if deps is not None else [], + }, + status=status.HTTP_400_BAD_REQUEST) + class CollectionItemChunkViewSet(viewsets.ViewSet): allowed_methods = ['GET', 'POST'] From 6b0a40e9dd87a35f716186904b36b263c0155d98 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 13 May 2020 16:01:49 +0300 Subject: [PATCH 120/511] Set custom user model and reset migrations. --- django_etesync/migrations/0001_initial.py | 52 +++++++++++++----- .../migrations/0002_auto_20200220_0943.py | 53 ------------------- .../migrations/0003_collectionitem_current.py | 18 ------- .../migrations/0004_auto_20200220_1029.py | 18 ------- .../migrations/0005_auto_20200220_1123.py | 29 ---------- .../migrations/0006_auto_20200220_1137.py | 49 ----------------- .../migrations/0007_auto_20200220_1144.py | 28 ---------- .../0008_collectionitemchunk_chunkfile.py | 20 ------- .../migrations/0009_auto_20200220_1220.py | 19 ------- .../migrations/0010_auto_20200220_1248.py | 19 ------- .../migrations/0011_auto_20200220_2037.py | 17 ------ .../migrations/0012_auto_20200220_2038.py | 19 ------- ...0013_collectionitemrevision_is_deletion.py | 18 ------- .../migrations/0014_auto_20200226_1322.py | 18 ------- .../migrations/0015_auto_20200226_1349.py | 18 ------- .../migrations/0016_auto_20200226_1446.py | 29 ---------- .../migrations/0017_auto_20200226_1455.py | 18 ------- .../migrations/0018_auto_20200226_1803.py | 18 ------- .../migrations/0019_collectionmember.py | 29 ---------- .../migrations/0020_auto_20200310_1438.py | 29 ---------- .../migrations/0021_auto_20200310_1439.py | 24 --------- .../migrations/0022_auto_20200310_1547.py | 33 ------------ .../migrations/0023_auto_20200310_1556.py | 17 ------ .../0024_collectionitemrevision_uid.py | 19 ------- .../migrations/0025_auto_20200312_1350.py | 19 ------- .../0026_collectionitemrevision_meta.py | 18 ------- .../migrations/0027_collection_mainitem.py | 19 ------- .../migrations/0028_auto_20200312_1819.py | 24 --------- .../migrations/0029_auto_20200312_1849.py | 35 ------------ .../migrations/0030_auto_20200312_1859.py | 19 ------- .../migrations/0031_auto_20200317_1509.py | 19 ------- .../migrations/0032_auto_20200317_1513.py | 23 -------- .../migrations/0033_auto_20200317_2010.py | 19 ------- .../migrations/0034_auto_20200415_1248.py | 19 ------- .../migrations/0035_auto_20200415_1259.py | 19 ------- .../migrations/0036_auto_20200415_1420.py | 37 ------------- .../migrations/0037_auto_20200415_1421.py | 23 -------- ...38_remove_collectionitemrevision_chunks.py | 17 ------ .../0039_remove_collection_mainitem.py | 17 ------ myauth/__init__.py | 0 myauth/admin.py | 5 ++ myauth/apps.py | 5 ++ myauth/migrations/0001_initial.py | 44 +++++++++++++++ myauth/migrations/__init__.py | 0 myauth/models.py | 5 ++ myauth/tests.py | 3 ++ myauth/views.py | 3 ++ 47 files changed, 103 insertions(+), 900 deletions(-) delete mode 100644 django_etesync/migrations/0002_auto_20200220_0943.py delete mode 100644 django_etesync/migrations/0003_collectionitem_current.py delete mode 100644 django_etesync/migrations/0004_auto_20200220_1029.py delete mode 100644 django_etesync/migrations/0005_auto_20200220_1123.py delete mode 100644 django_etesync/migrations/0006_auto_20200220_1137.py delete mode 100644 django_etesync/migrations/0007_auto_20200220_1144.py delete mode 100644 django_etesync/migrations/0008_collectionitemchunk_chunkfile.py delete mode 100644 django_etesync/migrations/0009_auto_20200220_1220.py delete mode 100644 django_etesync/migrations/0010_auto_20200220_1248.py delete mode 100644 django_etesync/migrations/0011_auto_20200220_2037.py delete mode 100644 django_etesync/migrations/0012_auto_20200220_2038.py delete mode 100644 django_etesync/migrations/0013_collectionitemrevision_is_deletion.py delete mode 100644 django_etesync/migrations/0014_auto_20200226_1322.py delete mode 100644 django_etesync/migrations/0015_auto_20200226_1349.py delete mode 100644 django_etesync/migrations/0016_auto_20200226_1446.py delete mode 100644 django_etesync/migrations/0017_auto_20200226_1455.py delete mode 100644 django_etesync/migrations/0018_auto_20200226_1803.py delete mode 100644 django_etesync/migrations/0019_collectionmember.py delete mode 100644 django_etesync/migrations/0020_auto_20200310_1438.py delete mode 100644 django_etesync/migrations/0021_auto_20200310_1439.py delete mode 100644 django_etesync/migrations/0022_auto_20200310_1547.py delete mode 100644 django_etesync/migrations/0023_auto_20200310_1556.py delete mode 100644 django_etesync/migrations/0024_collectionitemrevision_uid.py delete mode 100644 django_etesync/migrations/0025_auto_20200312_1350.py delete mode 100644 django_etesync/migrations/0026_collectionitemrevision_meta.py delete mode 100644 django_etesync/migrations/0027_collection_mainitem.py delete mode 100644 django_etesync/migrations/0028_auto_20200312_1819.py delete mode 100644 django_etesync/migrations/0029_auto_20200312_1849.py delete mode 100644 django_etesync/migrations/0030_auto_20200312_1859.py delete mode 100644 django_etesync/migrations/0031_auto_20200317_1509.py delete mode 100644 django_etesync/migrations/0032_auto_20200317_1513.py delete mode 100644 django_etesync/migrations/0033_auto_20200317_2010.py delete mode 100644 django_etesync/migrations/0034_auto_20200415_1248.py delete mode 100644 django_etesync/migrations/0035_auto_20200415_1259.py delete mode 100644 django_etesync/migrations/0036_auto_20200415_1420.py delete mode 100644 django_etesync/migrations/0037_auto_20200415_1421.py delete mode 100644 django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py delete mode 100644 django_etesync/migrations/0039_remove_collection_mainitem.py create mode 100644 myauth/__init__.py create mode 100644 myauth/admin.py create mode 100644 myauth/apps.py create mode 100644 myauth/migrations/0001_initial.py create mode 100644 myauth/migrations/__init__.py create mode 100644 myauth/models.py create mode 100644 myauth/tests.py create mode 100644 myauth/views.py diff --git a/django_etesync/migrations/0001_initial.py b/django_etesync/migrations/0001_initial.py index 6fa724b..dc3a2ff 100644 --- a/django_etesync/migrations/0001_initial.py +++ b/django_etesync/migrations/0001_initial.py @@ -1,9 +1,10 @@ -# Generated by Django 3.0.3 on 2020-02-19 15:33 +# Generated by Django 3.0.3 on 2020-05-13 13:01 from django.conf import settings import django.core.validators from django.db import migrations, models import django.db.models.deletion +import django_etesync.models class Migration(migrations.Migration): @@ -19,7 +20,7 @@ class Migration(migrations.Migration): name='Collection', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-fA-F0-9\\-_=]{44}')])), + ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')])), ('version', models.PositiveSmallIntegerField()), ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], @@ -31,37 +32,60 @@ class Migration(migrations.Migration): name='CollectionItem', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-fA-F0-9\\-_=]{44}')])), + ('uid', models.CharField(db_index=True, max_length=44, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')])), ('version', models.PositiveSmallIntegerField()), - ('encryptionKey', models.BinaryField(editable=True)), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etesync.Collection')), + ('encryptionKey', models.BinaryField(editable=True, null=True)), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='items', to='django_etesync.Collection')), ], options={ 'unique_together': {('uid', 'collection')}, }, ), migrations.CreateModel( - name='CollectionItemSnapshot', + name='CollectionItemChunk', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), + ('chunkFile', models.FileField(max_length=150, unique=True, upload_to=django_etesync.models.chunk_directory_path)), + ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etesync.CollectionItem')), + ], + ), + migrations.CreateModel( + name='CollectionItemRevision', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('current', models.BooleanField(default=True)), - ('chunkHmac', models.CharField(max_length=50)), - ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etesync.CollectionItem')), + ('uid', models.CharField(db_index=True, max_length=44, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), + ('meta', models.BinaryField(editable=True)), + ('current', models.BooleanField(db_index=True, default=True, null=True)), + ('deleted', models.BooleanField(default=False)), + ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='django_etesync.CollectionItem')), ], options={ 'unique_together': {('item', 'current')}, }, ), migrations.CreateModel( - name='CollectionItemChunk', + name='RevisionChunkRelation', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-fA-F0-9\\-_=]{44}')])), - ('order', models.CharField(max_length=100)), - ('itemSnapshot', models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, to='django_etesync.CollectionItemSnapshot')), + ('chunk', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions_relation', to='django_etesync.CollectionItemChunk')), + ('revision', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks_relation', to='django_etesync.CollectionItemRevision')), + ], + options={ + 'ordering': ('id',), + }, + ), + migrations.CreateModel( + name='CollectionMember', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('encryptionKey', models.BinaryField(editable=True)), + ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='members', to='django_etesync.Collection')), + ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], options={ - 'ordering': ['order'], + 'unique_together': {('user', 'collection')}, }, ), ] diff --git a/django_etesync/migrations/0002_auto_20200220_0943.py b/django_etesync/migrations/0002_auto_20200220_0943.py deleted file mode 100644 index c150a11..0000000 --- a/django_etesync/migrations/0002_auto_20200220_0943.py +++ /dev/null @@ -1,53 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 09:43 - -import django.core.validators -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0001_initial'), - ] - - operations = [ - migrations.RemoveField( - model_name='collectionitemchunk', - name='itemSnapshot', - ), - migrations.AddField( - model_name='collectionitem', - name='hmac', - field=models.CharField(default='', max_length=50), - preserve_default=False, - ), - migrations.AddField( - model_name='collectionitemchunk', - name='items', - field=models.ManyToManyField(related_name='chunks', to='django_etesync.CollectionItem'), - ), - migrations.AlterField( - model_name='collection', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{44}')]), - ), - migrations.AlterField( - model_name='collectionitem', - name='collection', - field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='items', to='django_etesync.Collection'), - ), - migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{44}')]), - ), - migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{44}')]), - ), - migrations.DeleteModel( - name='CollectionItemSnapshot', - ), - ] diff --git a/django_etesync/migrations/0003_collectionitem_current.py b/django_etesync/migrations/0003_collectionitem_current.py deleted file mode 100644 index 2ffbf54..0000000 --- a/django_etesync/migrations/0003_collectionitem_current.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 09:47 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0002_auto_20200220_0943'), - ] - - operations = [ - migrations.AddField( - model_name='collectionitem', - name='current', - field=models.BooleanField(default=True), - ), - ] diff --git a/django_etesync/migrations/0004_auto_20200220_1029.py b/django_etesync/migrations/0004_auto_20200220_1029.py deleted file mode 100644 index 1ea337a..0000000 --- a/django_etesync/migrations/0004_auto_20200220_1029.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 10:29 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0003_collectionitem_current'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitem', - name='current', - field=models.BooleanField(db_index=True, default=True), - ), - ] diff --git a/django_etesync/migrations/0005_auto_20200220_1123.py b/django_etesync/migrations/0005_auto_20200220_1123.py deleted file mode 100644 index 88c9ea6..0000000 --- a/django_etesync/migrations/0005_auto_20200220_1123.py +++ /dev/null @@ -1,29 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 11:23 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0004_auto_20200220_1029'), - ] - - operations = [ - migrations.RemoveField( - model_name='collectionitemchunk', - name='items', - ), - migrations.AddField( - model_name='collectionitem', - name='chunks', - field=models.ManyToManyField(related_name='items', to='django_etesync.CollectionItemChunk'), - ), - migrations.AddField( - model_name='collectionitemchunk', - name='collection', - field=models.ForeignKey(default=1, on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etesync.Collection'), - preserve_default=False, - ), - ] diff --git a/django_etesync/migrations/0006_auto_20200220_1137.py b/django_etesync/migrations/0006_auto_20200220_1137.py deleted file mode 100644 index efc421e..0000000 --- a/django_etesync/migrations/0006_auto_20200220_1137.py +++ /dev/null @@ -1,49 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 11:37 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0005_auto_20200220_1123'), - ] - - operations = [ - migrations.RemoveField( - model_name='collectionitem', - name='chunks', - ), - migrations.RemoveField( - model_name='collectionitem', - name='current', - ), - migrations.RemoveField( - model_name='collectionitem', - name='encryptionKey', - ), - migrations.RemoveField( - model_name='collectionitem', - name='hmac', - ), - migrations.RemoveField( - model_name='collectionitem', - name='version', - ), - migrations.CreateModel( - name='CollectionItemSnapshot', - fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('version', models.PositiveSmallIntegerField()), - ('encryptionKey', models.BinaryField(editable=True)), - ('hmac', models.CharField(max_length=50)), - ('current', models.BooleanField(db_index=True, default=True)), - ('chunks', models.ManyToManyField(related_name='items', to='django_etesync.CollectionItemChunk')), - ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='snapshots', to='django_etesync.CollectionItem')), - ], - options={ - 'unique_together': {('item', 'current')}, - }, - ), - ] diff --git a/django_etesync/migrations/0007_auto_20200220_1144.py b/django_etesync/migrations/0007_auto_20200220_1144.py deleted file mode 100644 index 3ebf55b..0000000 --- a/django_etesync/migrations/0007_auto_20200220_1144.py +++ /dev/null @@ -1,28 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 11:44 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0006_auto_20200220_1137'), - ] - - operations = [ - migrations.AddField( - model_name='collectionitemchunk', - name='item', - field=models.ForeignKey(default=1, on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etesync.CollectionItem'), - preserve_default=False, - ), - migrations.AlterUniqueTogether( - name='collectionitemchunk', - unique_together={('item', 'order')}, - ), - migrations.RemoveField( - model_name='collectionitemchunk', - name='collection', - ), - ] diff --git a/django_etesync/migrations/0008_collectionitemchunk_chunkfile.py b/django_etesync/migrations/0008_collectionitemchunk_chunkfile.py deleted file mode 100644 index 68bb27c..0000000 --- a/django_etesync/migrations/0008_collectionitemchunk_chunkfile.py +++ /dev/null @@ -1,20 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 12:16 - -from django.db import migrations, models -import django_etesync.models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0007_auto_20200220_1144'), - ] - - operations = [ - migrations.AddField( - model_name='collectionitemchunk', - name='chunkFile', - field=models.FileField(default='', upload_to=django_etesync.models.chunk_directory_path), - preserve_default=False, - ), - ] diff --git a/django_etesync/migrations/0009_auto_20200220_1220.py b/django_etesync/migrations/0009_auto_20200220_1220.py deleted file mode 100644 index 71e1539..0000000 --- a/django_etesync/migrations/0009_auto_20200220_1220.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 12:20 - -from django.db import migrations, models -import django_etesync.models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0008_collectionitemchunk_chunkfile'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemchunk', - name='chunkFile', - field=models.FileField(max_length=150, upload_to=django_etesync.models.chunk_directory_path), - ), - ] diff --git a/django_etesync/migrations/0010_auto_20200220_1248.py b/django_etesync/migrations/0010_auto_20200220_1248.py deleted file mode 100644 index 0c08ed0..0000000 --- a/django_etesync/migrations/0010_auto_20200220_1248.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 12:48 - -from django.db import migrations, models -import django_etesync.models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0009_auto_20200220_1220'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemchunk', - name='chunkFile', - field=models.FileField(max_length=150, unique=True, upload_to=django_etesync.models.chunk_directory_path), - ), - ] diff --git a/django_etesync/migrations/0011_auto_20200220_2037.py b/django_etesync/migrations/0011_auto_20200220_2037.py deleted file mode 100644 index 2d790748..0000000 --- a/django_etesync/migrations/0011_auto_20200220_2037.py +++ /dev/null @@ -1,17 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 20:37 - -from django.db import migrations - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0010_auto_20200220_1248'), - ] - - operations = [ - migrations.RenameModel( - old_name='CollectionItemSnapshot', - new_name='CollectionItemRevision', - ), - ] diff --git a/django_etesync/migrations/0012_auto_20200220_2038.py b/django_etesync/migrations/0012_auto_20200220_2038.py deleted file mode 100644 index 2657973..0000000 --- a/django_etesync/migrations/0012_auto_20200220_2038.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-20 20:38 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0011_auto_20200220_2037'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemrevision', - name='item', - field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='django_etesync.CollectionItem'), - ), - ] diff --git a/django_etesync/migrations/0013_collectionitemrevision_is_deletion.py b/django_etesync/migrations/0013_collectionitemrevision_is_deletion.py deleted file mode 100644 index 27f4953..0000000 --- a/django_etesync/migrations/0013_collectionitemrevision_is_deletion.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 13:17 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0012_auto_20200220_2038'), - ] - - operations = [ - migrations.AddField( - model_name='collectionitemrevision', - name='is_deletion', - field=models.BooleanField(default=False), - ), - ] diff --git a/django_etesync/migrations/0014_auto_20200226_1322.py b/django_etesync/migrations/0014_auto_20200226_1322.py deleted file mode 100644 index 1937015..0000000 --- a/django_etesync/migrations/0014_auto_20200226_1322.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 13:22 - -from django.db import migrations - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0013_collectionitemrevision_is_deletion'), - ] - - operations = [ - migrations.RenameField( - model_name='collectionitemrevision', - old_name='is_deletion', - new_name='isDeletion', - ), - ] diff --git a/django_etesync/migrations/0015_auto_20200226_1349.py b/django_etesync/migrations/0015_auto_20200226_1349.py deleted file mode 100644 index 896619d..0000000 --- a/django_etesync/migrations/0015_auto_20200226_1349.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 13:49 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0014_auto_20200226_1322'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemrevision', - name='current', - field=models.BooleanField(blank=True, db_index=True, default=True, null=True), - ), - ] diff --git a/django_etesync/migrations/0016_auto_20200226_1446.py b/django_etesync/migrations/0016_auto_20200226_1446.py deleted file mode 100644 index 2929cbf..0000000 --- a/django_etesync/migrations/0016_auto_20200226_1446.py +++ /dev/null @@ -1,29 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 14:46 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0015_auto_20200226_1349'), - ] - - operations = [ - migrations.AlterField( - model_name='collection', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{43}')]), - ), - migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{43}')]), - ), - migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID. Expected a 256bit base64url.', regex='[a-zA-Z0-9\\-_=]{43}')]), - ), - ] diff --git a/django_etesync/migrations/0017_auto_20200226_1455.py b/django_etesync/migrations/0017_auto_20200226_1455.py deleted file mode 100644 index 2148123..0000000 --- a/django_etesync/migrations/0017_auto_20200226_1455.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 14:55 - -from django.db import migrations - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0016_auto_20200226_1446'), - ] - - operations = [ - migrations.RenameField( - model_name='collectionitemrevision', - old_name='isDeletion', - new_name='deleted', - ), - ] diff --git a/django_etesync/migrations/0018_auto_20200226_1803.py b/django_etesync/migrations/0018_auto_20200226_1803.py deleted file mode 100644 index ae9200d..0000000 --- a/django_etesync/migrations/0018_auto_20200226_1803.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 18:03 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0017_auto_20200226_1455'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemrevision', - name='current', - field=models.BooleanField(db_index=True, default=True, null=True), - ), - ] diff --git a/django_etesync/migrations/0019_collectionmember.py b/django_etesync/migrations/0019_collectionmember.py deleted file mode 100644 index 142e945..0000000 --- a/django_etesync/migrations/0019_collectionmember.py +++ /dev/null @@ -1,29 +0,0 @@ -# Generated by Django 3.0.3 on 2020-02-26 18:33 - -from django.conf import settings -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - migrations.swappable_dependency(settings.AUTH_USER_MODEL), - ('django_etesync', '0018_auto_20200226_1803'), - ] - - operations = [ - migrations.CreateModel( - name='CollectionMember', - fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('encryptionKey', models.BinaryField(editable=True)), - ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='members', to='django_etesync.Collection')), - ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), - ], - options={ - 'unique_together': {('user', 'collection')}, - }, - ), - ] diff --git a/django_etesync/migrations/0020_auto_20200310_1438.py b/django_etesync/migrations/0020_auto_20200310_1438.py deleted file mode 100644 index 6949145..0000000 --- a/django_etesync/migrations/0020_auto_20200310_1438.py +++ /dev/null @@ -1,29 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-10 14:38 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0019_collectionmember'), - ] - - operations = [ - migrations.AlterField( - model_name='collection', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]{24}')]), - ), - migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]{24}')]), - ), - migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}=?$')]), - ), - ] diff --git a/django_etesync/migrations/0021_auto_20200310_1439.py b/django_etesync/migrations/0021_auto_20200310_1439.py deleted file mode 100644 index 3f1341e..0000000 --- a/django_etesync/migrations/0021_auto_20200310_1439.py +++ /dev/null @@ -1,24 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-10 14:39 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0020_auto_20200310_1438'), - ] - - operations = [ - migrations.AlterField( - model_name='collection', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), - ), - migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), - ), - ] diff --git a/django_etesync/migrations/0022_auto_20200310_1547.py b/django_etesync/migrations/0022_auto_20200310_1547.py deleted file mode 100644 index cbd0ee7..0000000 --- a/django_etesync/migrations/0022_auto_20200310_1547.py +++ /dev/null @@ -1,33 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-10 15:47 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0021_auto_20200310_1439'), - ] - - operations = [ - migrations.RemoveField( - model_name='collectionitemrevision', - name='encryptionKey', - ), - migrations.RemoveField( - model_name='collectionitemrevision', - name='version', - ), - migrations.AddField( - model_name='collectionitem', - name='encryptionKey', - field=models.BinaryField(default=b'aoesnutheounth', editable=True), - preserve_default=False, - ), - migrations.AddField( - model_name='collectionitem', - name='version', - field=models.PositiveSmallIntegerField(default=1), - preserve_default=False, - ), - ] diff --git a/django_etesync/migrations/0023_auto_20200310_1556.py b/django_etesync/migrations/0023_auto_20200310_1556.py deleted file mode 100644 index e2a9b80..0000000 --- a/django_etesync/migrations/0023_auto_20200310_1556.py +++ /dev/null @@ -1,17 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-10 15:56 - -from django.db import migrations - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0022_auto_20200310_1547'), - ] - - operations = [ - migrations.AlterModelOptions( - name='collectionitemchunk', - options={'ordering': ('item', 'order')}, - ), - ] diff --git a/django_etesync/migrations/0024_collectionitemrevision_uid.py b/django_etesync/migrations/0024_collectionitemrevision_uid.py deleted file mode 100644 index 6134c89..0000000 --- a/django_etesync/migrations/0024_collectionitemrevision_uid.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 13:41 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0023_auto_20200310_1556'), - ] - - operations = [ - migrations.AddField( - model_name='collectionitemrevision', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), - ), - ] diff --git a/django_etesync/migrations/0025_auto_20200312_1350.py b/django_etesync/migrations/0025_auto_20200312_1350.py deleted file mode 100644 index b54aeff..0000000 --- a/django_etesync/migrations/0025_auto_20200312_1350.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 13:50 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0024_collectionitemrevision_uid'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemrevision', - name='uid', - field=models.CharField(db_index=True, max_length=44, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), - ), - ] diff --git a/django_etesync/migrations/0026_collectionitemrevision_meta.py b/django_etesync/migrations/0026_collectionitemrevision_meta.py deleted file mode 100644 index 8056e61..0000000 --- a/django_etesync/migrations/0026_collectionitemrevision_meta.py +++ /dev/null @@ -1,18 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 14:01 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0025_auto_20200312_1350'), - ] - - operations = [ - migrations.AddField( - model_name='collectionitemrevision', - name='meta', - field=models.BinaryField(blank=True, editable=True, null=True), - ), - ] diff --git a/django_etesync/migrations/0027_collection_mainitem.py b/django_etesync/migrations/0027_collection_mainitem.py deleted file mode 100644 index b420d8f..0000000 --- a/django_etesync/migrations/0027_collection_mainitem.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 14:14 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0026_collectionitemrevision_meta'), - ] - - operations = [ - migrations.AddField( - model_name='collection', - name='mainItem', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, related_name='of_collection', to='django_etesync.CollectionItem'), - ), - ] diff --git a/django_etesync/migrations/0028_auto_20200312_1819.py b/django_etesync/migrations/0028_auto_20200312_1819.py deleted file mode 100644 index 6d76499..0000000 --- a/django_etesync/migrations/0028_auto_20200312_1819.py +++ /dev/null @@ -1,24 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 18:19 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0027_collection_mainitem'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitem', - name='encryptionKey', - field=models.BinaryField(editable=True, null=True), - ), - migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=44, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')]), - ), - ] diff --git a/django_etesync/migrations/0029_auto_20200312_1849.py b/django_etesync/migrations/0029_auto_20200312_1849.py deleted file mode 100644 index 568aaa2..0000000 --- a/django_etesync/migrations/0029_auto_20200312_1849.py +++ /dev/null @@ -1,35 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 18:49 - -from django.db import migrations -from django.utils.crypto import get_random_string - - -def generate_rev_uid(length=32): - return get_random_string(length) - - -def add_collection_main_item(apps, schema_editor): - Collection = apps.get_model('django_etesync', 'Collection') - CollectionItem = apps.get_model('django_etesync', 'CollectionItem') - CollectionItemRevision = apps.get_model('django_etesync', 'CollectionItemRevision') - - for col in Collection.objects.all(): - main_item = CollectionItem.objects.create(uid=None, encryptionKey=None, version=col.version, collection=col) - col.mainItem = main_item - col.save() - - CollectionItemRevision.objects.create( - uid=generate_rev_uid(), - hmac='hmac-hash', - item=main_item) - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0028_auto_20200312_1819'), - ] - - operations = [ - migrations.RunPython(add_collection_main_item), - ] diff --git a/django_etesync/migrations/0030_auto_20200312_1859.py b/django_etesync/migrations/0030_auto_20200312_1859.py deleted file mode 100644 index fe8050a..0000000 --- a/django_etesync/migrations/0030_auto_20200312_1859.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-12 18:59 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0029_auto_20200312_1849'), - ] - - operations = [ - migrations.AlterField( - model_name='collection', - name='mainItem', - field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, related_name='of_collection', to='django_etesync.CollectionItem'), - ), - ] diff --git a/django_etesync/migrations/0031_auto_20200317_1509.py b/django_etesync/migrations/0031_auto_20200317_1509.py deleted file mode 100644 index 7166781..0000000 --- a/django_etesync/migrations/0031_auto_20200317_1509.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-17 15:09 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0030_auto_20200312_1859'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')]), - ), - ] diff --git a/django_etesync/migrations/0032_auto_20200317_1513.py b/django_etesync/migrations/0032_auto_20200317_1513.py deleted file mode 100644 index 0546711..0000000 --- a/django_etesync/migrations/0032_auto_20200317_1513.py +++ /dev/null @@ -1,23 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-17 15:13 - -import django.core.validators -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0031_auto_20200317_1509'), - ] - - operations = [ - migrations.RemoveField( - model_name='collectionitemrevision', - name='hmac', - ), - migrations.AlterField( - model_name='collectionitemrevision', - name='uid', - field=models.CharField(db_index=True, max_length=44, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')]), - ), - ] diff --git a/django_etesync/migrations/0033_auto_20200317_2010.py b/django_etesync/migrations/0033_auto_20200317_2010.py deleted file mode 100644 index 7a42b38..0000000 --- a/django_etesync/migrations/0033_auto_20200317_2010.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-03-17 20:10 - -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0032_auto_20200317_1513'), - ] - - operations = [ - migrations.AlterField( - model_name='collectionitemrevision', - name='meta', - field=models.BinaryField(default=b'', editable=True), - preserve_default=False, - ), - ] diff --git a/django_etesync/migrations/0034_auto_20200415_1248.py b/django_etesync/migrations/0034_auto_20200415_1248.py deleted file mode 100644 index 1156676..0000000 --- a/django_etesync/migrations/0034_auto_20200415_1248.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-04-15 12:48 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0033_auto_20200317_2010'), - ] - - operations = [ - migrations.AlterField( - model_name='collection', - name='mainItem', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, related_name='of_collection', to='django_etesync.CollectionItem'), - ), - ] diff --git a/django_etesync/migrations/0035_auto_20200415_1259.py b/django_etesync/migrations/0035_auto_20200415_1259.py deleted file mode 100644 index d558e31..0000000 --- a/django_etesync/migrations/0035_auto_20200415_1259.py +++ /dev/null @@ -1,19 +0,0 @@ -# Generated by Django 3.0.3 on 2020-04-15 12:59 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0034_auto_20200415_1248'), - ] - - operations = [ - migrations.AlterField( - model_name='collection', - name='mainItem', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='of_collection', to='django_etesync.CollectionItem'), - ), - ] diff --git a/django_etesync/migrations/0036_auto_20200415_1420.py b/django_etesync/migrations/0036_auto_20200415_1420.py deleted file mode 100644 index a7b8003..0000000 --- a/django_etesync/migrations/0036_auto_20200415_1420.py +++ /dev/null @@ -1,37 +0,0 @@ -# Generated by Django 3.0.3 on 2020-04-15 14:20 - -from django.db import migrations, models -import django.db.models.deletion - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0035_auto_20200415_1259'), - ] - - operations = [ - migrations.AlterModelOptions( - name='collectionitemchunk', - options={}, - ), - migrations.AlterUniqueTogether( - name='collectionitemchunk', - unique_together=set(), - ), - migrations.CreateModel( - name='RevisionChunkRelation', - fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('chunk', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions_relation', to='django_etesync.CollectionItemChunk')), - ('revision', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks_relation', to='django_etesync.CollectionItemRevision')), - ], - options={ - 'ordering': ('id',), - }, - ), - migrations.RemoveField( - model_name='collectionitemchunk', - name='order', - ), - ] diff --git a/django_etesync/migrations/0037_auto_20200415_1421.py b/django_etesync/migrations/0037_auto_20200415_1421.py deleted file mode 100644 index d1a47db..0000000 --- a/django_etesync/migrations/0037_auto_20200415_1421.py +++ /dev/null @@ -1,23 +0,0 @@ -# Generated by Django 3.0.3 on 2020-04-15 14:21 - -from django.db import migrations - - -def change_chunk_relation(apps, schema_editor): - CollectionItemRevision = apps.get_model('django_etesync', 'CollectionItemRevision') - RevisionChunkRelation = apps.get_model('django_etesync', 'RevisionChunkRelation') - - for revision in CollectionItemRevision.objects.all(): - for chunk in revision.chunks.all(): - RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0036_auto_20200415_1420'), - ] - - operations = [ - migrations.RunPython(change_chunk_relation), - ] diff --git a/django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py b/django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py deleted file mode 100644 index 6e35b86..0000000 --- a/django_etesync/migrations/0038_remove_collectionitemrevision_chunks.py +++ /dev/null @@ -1,17 +0,0 @@ -# Generated by Django 3.0.3 on 2020-04-15 14:34 - -from django.db import migrations - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0037_auto_20200415_1421'), - ] - - operations = [ - migrations.RemoveField( - model_name='collectionitemrevision', - name='chunks', - ), - ] diff --git a/django_etesync/migrations/0039_remove_collection_mainitem.py b/django_etesync/migrations/0039_remove_collection_mainitem.py deleted file mode 100644 index 1822bc7..0000000 --- a/django_etesync/migrations/0039_remove_collection_mainitem.py +++ /dev/null @@ -1,17 +0,0 @@ -# Generated by Django 3.0.3 on 2020-04-16 08:28 - -from django.db import migrations - - -class Migration(migrations.Migration): - - dependencies = [ - ('django_etesync', '0038_remove_collectionitemrevision_chunks'), - ] - - operations = [ - migrations.RemoveField( - model_name='collection', - name='mainItem', - ), - ] diff --git a/myauth/__init__.py b/myauth/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/myauth/admin.py b/myauth/admin.py new file mode 100644 index 0000000..f91be8f --- /dev/null +++ b/myauth/admin.py @@ -0,0 +1,5 @@ +from django.contrib import admin +from django.contrib.auth.admin import UserAdmin +from .models import User + +admin.site.register(User, UserAdmin) diff --git a/myauth/apps.py b/myauth/apps.py new file mode 100644 index 0000000..611e83d --- /dev/null +++ b/myauth/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class MyauthConfig(AppConfig): + name = 'myauth' diff --git a/myauth/migrations/0001_initial.py b/myauth/migrations/0001_initial.py new file mode 100644 index 0000000..1f81e95 --- /dev/null +++ b/myauth/migrations/0001_initial.py @@ -0,0 +1,44 @@ +# Generated by Django 3.0.3 on 2020-05-13 13:00 + +import django.contrib.auth.models +import django.contrib.auth.validators +from django.db import migrations, models +import django.utils.timezone + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ('auth', '0011_update_proxy_permissions'), + ] + + operations = [ + migrations.CreateModel( + name='User', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('password', models.CharField(max_length=128, verbose_name='password')), + ('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')), + ('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')), + ('username', models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150, unique=True, validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], verbose_name='username')), + ('first_name', models.CharField(blank=True, max_length=30, verbose_name='first name')), + ('last_name', models.CharField(blank=True, max_length=150, verbose_name='last name')), + ('email', models.EmailField(blank=True, max_length=254, verbose_name='email address')), + ('is_staff', models.BooleanField(default=False, help_text='Designates whether the user can log into this admin site.', verbose_name='staff status')), + ('is_active', models.BooleanField(default=True, help_text='Designates whether this user should be treated as active. Unselect this instead of deleting accounts.', verbose_name='active')), + ('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='date joined')), + ('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.Group', verbose_name='groups')), + ('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.Permission', verbose_name='user permissions')), + ], + options={ + 'verbose_name': 'user', + 'verbose_name_plural': 'users', + 'abstract': False, + }, + managers=[ + ('objects', django.contrib.auth.models.UserManager()), + ], + ), + ] diff --git a/myauth/migrations/__init__.py b/myauth/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/myauth/models.py b/myauth/models.py new file mode 100644 index 0000000..3d30525 --- /dev/null +++ b/myauth/models.py @@ -0,0 +1,5 @@ +from django.contrib.auth.models import AbstractUser + + +class User(AbstractUser): + pass diff --git a/myauth/tests.py b/myauth/tests.py new file mode 100644 index 0000000..7ce503c --- /dev/null +++ b/myauth/tests.py @@ -0,0 +1,3 @@ +from django.test import TestCase + +# Create your tests here. diff --git a/myauth/views.py b/myauth/views.py new file mode 100644 index 0000000..91ea44a --- /dev/null +++ b/myauth/views.py @@ -0,0 +1,3 @@ +from django.shortcuts import render + +# Create your views here. From 32a8b9c90dc798d20c4e7faaee6fc39d220675ed Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 14 May 2020 13:43:49 +0300 Subject: [PATCH 121/511] Implement a ZKPP login flow. --- django_etesync/app_settings.py | 4 + django_etesync/migrations/0002_userinfo.py | 25 +++++ django_etesync/models.py | 10 ++ django_etesync/serializers.py | 64 ++++++++++++ django_etesync/views.py | 116 +++++++++++++++++++++ requirements.in/base.txt | 1 + requirements.txt | 27 ++--- 7 files changed, 235 insertions(+), 12 deletions(-) create mode 100644 django_etesync/migrations/0002_userinfo.py diff --git a/django_etesync/app_settings.py b/django_etesync/app_settings.py index a0a0d99..89b38f7 100644 --- a/django_etesync/app_settings.py +++ b/django_etesync/app_settings.py @@ -46,5 +46,9 @@ class AppSettings: ret.append(self.import_from_str(perm)) return ret + @property + def CHALLENGE_VALID_SECONDS(self): # pylint: disable=invalid-name + return self._setting("CHALLENGE_VALID_SECONDS", 60) + app_settings = AppSettings('ETESYNC_') diff --git a/django_etesync/migrations/0002_userinfo.py b/django_etesync/migrations/0002_userinfo.py new file mode 100644 index 0000000..ad7018a --- /dev/null +++ b/django_etesync/migrations/0002_userinfo.py @@ -0,0 +1,25 @@ +# Generated by Django 3.0.3 on 2020-05-14 09:51 + +from django.conf import settings +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('myauth', '0001_initial'), + ('django_etesync', '0001_initial'), + ] + + operations = [ + migrations.CreateModel( + name='UserInfo', + fields=[ + ('owner', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, primary_key=True, serialize=False, to=settings.AUTH_USER_MODEL)), + ('version', models.PositiveSmallIntegerField(default=1)), + ('pubkey', models.BinaryField(editable=True)), + ('salt', models.BinaryField(editable=True)), + ], + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 05b81b3..0d0f3dc 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -137,3 +137,13 @@ class CollectionMember(models.Model): def __str__(self): return '{} {}'.format(self.collection.uid, self.user) + + +class UserInfo(models.Model): + owner = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, primary_key=True) + version = models.PositiveSmallIntegerField(default=1) + pubkey = models.BinaryField(editable=True, blank=False, null=False) + salt = models.BinaryField(editable=True, blank=False, null=False) + + def __str__(self): + return "UserInfo<{}>".format(self.owner) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index eb50d76..3497d78 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -198,3 +198,67 @@ class CollectionSerializer(serializers.ModelSerializer): process_revisions_for_item(main_item, revision_data) return instance + + +class UserSerializer(serializers.ModelSerializer): + class Meta: + model = User + fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) + + +class AuthenticationSignupSerializer(serializers.Serializer): + user = UserSerializer(many=False) + salt = BinaryBase64Field() + pubkey = BinaryBase64Field() + + def create(self, validated_data): + """Function that's called when this serializer creates an item""" + salt = validated_data.pop('salt') + pubkey = validated_data.pop('pubkey') + + with transaction.atomic(): + instance = UserSerializer.Meta.model.objects.create(**validated_data) + instance.set_unusable_password() + + models.UserInfo.objects.create(salt=salt, pubkey=pubkey, owner=instance) + + return instance + + def update(self, instance, validated_data): + raise NotImplementedError() + + +class AuthenticationLoginChallengeSerializer(serializers.Serializer): + username = serializers.CharField(required=False) + email = serializers.EmailField(required=False) + + def validate(self, data): + if not data.get('email') and not data.get('username'): + raise serializers.ValidationError('Either email or username must be set') + return data + + def create(self, validated_data): + raise NotImplementedError() + + def update(self, instance, validated_data): + raise NotImplementedError() + + +class AuthenticationLoginSerializer(AuthenticationLoginChallengeSerializer): + challenge = BinaryBase64Field() + host = serializers.CharField() + signature = BinaryBase64Field() + + def validate(self, data): + host = self.context.get('host', None) + if data['host'] != host: + raise serializers.ValidationError( + 'Found wrong host name. Got: "{}" expected: "{}"'.format(data['host'], host)) + + return super().validate(data) + + def create(self, validated_data): + raise NotImplementedError() + + def update(self, instance, validated_data): + raise NotImplementedError() diff --git a/django_etesync/views.py b/django_etesync/views.py index 87c8010..c628ab0 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -12,6 +12,8 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . +import json + from django.conf import settings from django.contrib.auth import get_user_model from django.db import transaction, IntegrityError @@ -24,10 +26,20 @@ from rest_framework import viewsets from rest_framework import parsers from rest_framework.decorators import action as action_decorator from rest_framework.response import Response +from rest_framework.authtoken.models import Token + +import nacl.encoding +import nacl.signing +import nacl.secret +import nacl.hash from . import app_settings from .models import Collection, CollectionItem, CollectionItemRevision from .serializers import ( + b64encode, + AuthenticationSignupSerializer, + AuthenticationLoginChallengeSerializer, + AuthenticationLoginSerializer, CollectionSerializer, CollectionItemSerializer, CollectionItemRevisionSerializer, @@ -290,6 +302,110 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return serve(request, basename, dirname) +class AuthenticationViewSet(viewsets.ViewSet): + allowed_methods = ['POST'] + + def get_encryption_key(self, salt): + key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) + return nacl.hash.blake2b(b'', key=key, salt=salt, person=b'etesync-auth', encoder=nacl.encoding.RawEncoder) + + def get_queryset(self): + return User.objects.all() + + def list(self, request): + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + @action_decorator(detail=False, methods=['POST']) + def signup(self, request): + serializer = AuthenticationSignupSerializer(data=request.data) + if serializer.is_valid(): + serializer.save() + + return Response({}, status=status.HTTP_201_CREATED) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + def get_login_user(self, serializer): + username = serializer.validated_data.get('username') + email = serializer.validated_data.get('email') + if username: + kwargs = {User.USERNAME_FIELD: username} + user = get_object_or_404(self.get_queryset(), **kwargs) + elif email: + kwargs = {User.EMAIL_FIELD: email} + user = get_object_or_404(self.get_queryset(), **kwargs) + + return user + + @action_decorator(detail=False, methods=['POST']) + def login_challenge(self, request): + from datetime import datetime + + serializer = AuthenticationLoginChallengeSerializer(data=request.data) + if serializer.is_valid(): + user = self.get_login_user(serializer) + + salt = user.userinfo.salt + enc_key = self.get_encryption_key(salt) + box = nacl.secret.SecretBox(enc_key) + + challenge_data = { + "timestamp": int(datetime.now().timestamp()), + "userId": user.id, + } + challenge = box.encrypt(json.dumps( + challenge_data, separators=(',', ':')).encode(), encoder=nacl.encoding.RawEncoder) + + ret = { + "salt": b64encode(salt), + "challenge": b64encode(challenge), + } + return Response(ret, status=status.HTTP_200_OK) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + @action_decorator(detail=False, methods=['POST']) + def login(self, request): + from datetime import datetime + + serializer = AuthenticationLoginSerializer( + data=request.data, context={'host': request.get_host()}) + if serializer.is_valid(): + user = self.get_login_user(serializer) + challenge = serializer.validated_data['challenge'] + signature = serializer.validated_data['signature'] + + salt = user.userinfo.salt + enc_key = self.get_encryption_key(salt) + box = nacl.secret.SecretBox(enc_key) + + challenge_data = json.loads(box.decrypt(challenge).decode()) + now = int(datetime.now().timestamp()) + if now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: + content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif challenge_data['userId'] != user.id: + content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + host_hash = nacl.hash.blake2b( + serializer.validated_data['host'].encode(), encoder=nacl.encoding.RawEncoder) + verify_key = nacl.signing.VerifyKey(user.userinfo.pubkey, encoder=nacl.encoding.RawEncoder) + verify_key.verify(challenge + host_hash, signature) + + data = { + 'token': Token.objects.get_or_create(user=user)[0].key, + } + return Response(data, status=status.HTTP_200_OK) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + + @action_decorator(detail=False, methods=['POST']) + def logout(self, request): + # FIXME: expire the token - we need better token handling - using knox? Something else? + return Response({}, status=status.HTTP_400_BAD_REQUEST) + + class ResetViewSet(BaseViewSet): allowed_methods = ['POST'] diff --git a/requirements.in/base.txt b/requirements.in/base.txt index a7d1734..e6d6379 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -9,3 +9,4 @@ django-ipware djangorestframework drf-nested-routers psycopg2-binary +pynacl diff --git a/requirements.txt b/requirements.txt index 34ca428..dcc37cf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,25 +6,28 @@ # asgiref==3.2.3 # via django certifi==2019.11.28 # via requests +cffi==1.14.0 # via pynacl chardet==3.0.4 # via requests defusedxml==0.6.0 # via python3-openid -django-allauth==0.41.0 -django-anymail==7.0.0 -django-appconf==1.0.3 -django-cors-headers==3.2.1 -django-debug-toolbar==2.2 -django-fullurl==1.0 -django-ipware==2.1.0 -django==3.0.3 -djangorestframework==3.11.0 -drf-nested-routers==0.91 +django-allauth==0.41.0 # via -r requirements.in/base.txt +django-anymail==7.0.0 # via -r requirements.in/base.txt +django-appconf==1.0.3 # via -r requirements.in/base.txt +django-cors-headers==3.2.1 # via -r requirements.in/base.txt +django-debug-toolbar==2.2 # via -r requirements.in/base.txt +django-fullurl==1.0 # via -r requirements.in/base.txt +django-ipware==2.1.0 # via -r requirements.in/base.txt +django==3.0.3 # via -r requirements.in/base.txt, django-allauth, django-anymail, django-appconf, django-cors-headers, django-debug-toolbar, django-fullurl, djangorestframework, drf-nested-routers +djangorestframework==3.11.0 # via -r requirements.in/base.txt, drf-nested-routers +drf-nested-routers==0.91 # via -r requirements.in/base.txt idna==2.8 # via requests oauthlib==3.1.0 # via requests-oauthlib -psycopg2-binary==2.8.4 +psycopg2-binary==2.8.4 # via -r requirements.in/base.txt +pycparser==2.20 # via cffi +pynacl==1.3.0 # via -r requirements.in/base.txt python3-openid==3.1.0 # via django-allauth pytz==2019.3 # via django requests-oauthlib==1.3.0 # via django-allauth requests==2.22.0 # via django-allauth, django-anymail, requests-oauthlib -six==1.14.0 # via django-anymail, django-appconf +six==1.14.0 # via django-anymail, django-appconf, pynacl sqlparse==0.3.0 # via django, django-debug-toolbar urllib3==1.25.8 # via requests From 93a0e41f03a937722da7d50b9006384f52b64815 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 14 May 2020 15:42:42 +0300 Subject: [PATCH 122/511] Change login flow to better verify all relevant fields. --- django_etesync/serializers.py | 20 ++++++----- django_etesync/views.py | 66 ++++++++++++++++++++--------------- 2 files changed, 48 insertions(+), 38 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 3497d78..fabc7cb 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -244,18 +244,20 @@ class AuthenticationLoginChallengeSerializer(serializers.Serializer): raise NotImplementedError() -class AuthenticationLoginSerializer(AuthenticationLoginChallengeSerializer): - challenge = BinaryBase64Field() - host = serializers.CharField() +class AuthenticationLoginSerializer(serializers.Serializer): + response = BinaryBase64Field() signature = BinaryBase64Field() - def validate(self, data): - host = self.context.get('host', None) - if data['host'] != host: - raise serializers.ValidationError( - 'Found wrong host name. Got: "{}" expected: "{}"'.format(data['host'], host)) + def create(self, validated_data): + raise NotImplementedError() - return super().validate(data) + def update(self, instance, validated_data): + raise NotImplementedError() + + +class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer): + challenge = BinaryBase64Field() + host = serializers.CharField() def create(self, validated_data): raise NotImplementedError() diff --git a/django_etesync/views.py b/django_etesync/views.py index c628ab0..cb52ca5 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -40,6 +40,7 @@ from .serializers import ( AuthenticationSignupSerializer, AuthenticationLoginChallengeSerializer, AuthenticationLoginSerializer, + AuthenticationLoginInnerSerializer, CollectionSerializer, CollectionItemSerializer, CollectionItemRevisionSerializer, @@ -368,35 +369,42 @@ class AuthenticationViewSet(viewsets.ViewSet): def login(self, request): from datetime import datetime - serializer = AuthenticationLoginSerializer( - data=request.data, context={'host': request.get_host()}) - if serializer.is_valid(): - user = self.get_login_user(serializer) - challenge = serializer.validated_data['challenge'] - signature = serializer.validated_data['signature'] - - salt = user.userinfo.salt - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = json.loads(box.decrypt(challenge).decode()) - now = int(datetime.now().timestamp()) - if now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: - content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif challenge_data['userId'] != user.id: - content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - host_hash = nacl.hash.blake2b( - serializer.validated_data['host'].encode(), encoder=nacl.encoding.RawEncoder) - verify_key = nacl.signing.VerifyKey(user.userinfo.pubkey, encoder=nacl.encoding.RawEncoder) - verify_key.verify(challenge + host_hash, signature) - - data = { - 'token': Token.objects.get_or_create(user=user)[0].key, - } - return Response(data, status=status.HTTP_200_OK) + outer_serializer = AuthenticationLoginSerializer(data=request.data) + if outer_serializer.is_valid(): + response_raw = outer_serializer.validated_data['response'] + response = json.loads(response_raw.decode()) + signature = outer_serializer.validated_data['signature'] + + serializer = AuthenticationLoginInnerSerializer(data=response, context={'host': request.get_host()}) + if serializer.is_valid(): + user = self.get_login_user(serializer) + host = serializer.validated_data['host'] + challenge = serializer.validated_data['challenge'] + + salt = user.userinfo.salt + enc_key = self.get_encryption_key(salt) + box = nacl.secret.SecretBox(enc_key) + + challenge_data = json.loads(box.decrypt(challenge).decode()) + now = int(datetime.now().timestamp()) + if now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: + content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif challenge_data['userId'] != user.id: + content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif host != request.get_host(): + detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) + content = {'code': 'wrong_host', 'detail': detail} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + verify_key = nacl.signing.VerifyKey(user.userinfo.pubkey, encoder=nacl.encoding.RawEncoder) + verify_key.verify(response_raw, signature) + + data = { + 'token': Token.objects.get_or_create(user=user)[0].key, + } + return Response(data, status=status.HTTP_200_OK) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) From e9e77945a6fb1723a1bae51a4eeeb865670bcf74 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 14 May 2020 17:19:18 +0300 Subject: [PATCH 123/511] Return token and user on signup/login. --- django_etesync/serializers.py | 10 ++++++++-- django_etesync/views.py | 19 +++++++++++++------ 2 files changed, 21 insertions(+), 8 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index fabc7cb..7173d73 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -206,8 +206,14 @@ class UserSerializer(serializers.ModelSerializer): fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) +class UserQuerySerializer(serializers.ModelSerializer): + class Meta: + model = User + fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) + + class AuthenticationSignupSerializer(serializers.Serializer): - user = UserSerializer(many=False) + user = UserQuerySerializer(many=False) salt = BinaryBase64Field() pubkey = BinaryBase64Field() @@ -217,7 +223,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): pubkey = validated_data.pop('pubkey') with transaction.atomic(): - instance = UserSerializer.Meta.model.objects.create(**validated_data) + instance = User.objects.create(**validated_data) instance.set_unusable_password() models.UserInfo.objects.create(salt=salt, pubkey=pubkey, owner=instance) diff --git a/django_etesync/views.py b/django_etesync/views.py index cb52ca5..eaf0e35 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -44,7 +44,8 @@ from .serializers import ( CollectionSerializer, CollectionItemSerializer, CollectionItemRevisionSerializer, - CollectionItemChunkSerializer + CollectionItemChunkSerializer, + UserSerializer, ) @@ -313,6 +314,12 @@ class AuthenticationViewSet(viewsets.ViewSet): def get_queryset(self): return User.objects.all() + def login_response_data(self, user): + return { + 'token': Token.objects.get_or_create(user=user)[0].key, + 'user': UserSerializer(user).data, + } + def list(self, request): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) @@ -320,9 +327,10 @@ class AuthenticationViewSet(viewsets.ViewSet): def signup(self, request): serializer = AuthenticationSignupSerializer(data=request.data) if serializer.is_valid(): - serializer.save() + user = serializer.save() - return Response({}, status=status.HTTP_201_CREATED) + data = self.login_response_data(user) + return Response(data, status=status.HTTP_201_CREATED) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) @@ -360,6 +368,7 @@ class AuthenticationViewSet(viewsets.ViewSet): ret = { "salt": b64encode(salt), "challenge": b64encode(challenge), + "version": user.userinfo.version, } return Response(ret, status=status.HTTP_200_OK) @@ -401,9 +410,7 @@ class AuthenticationViewSet(viewsets.ViewSet): verify_key = nacl.signing.VerifyKey(user.userinfo.pubkey, encoder=nacl.encoding.RawEncoder) verify_key.verify(response_raw, signature) - data = { - 'token': Token.objects.get_or_create(user=user)[0].key, - } + data = self.login_response_data(user) return Response(data, status=status.HTTP_200_OK) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) From 4083be8e8cb1412f8fff33062b660a67e9187f28 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 15 May 2020 11:01:56 +0300 Subject: [PATCH 124/511] Username: disallow @ in usernames. --- myauth/migrations/0002_auto_20200515_0801.py | 19 ++++++++++++++ myauth/models.py | 27 +++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 myauth/migrations/0002_auto_20200515_0801.py diff --git a/myauth/migrations/0002_auto_20200515_0801.py b/myauth/migrations/0002_auto_20200515_0801.py new file mode 100644 index 0000000..3ce02b2 --- /dev/null +++ b/myauth/migrations/0002_auto_20200515_0801.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-05-15 08:01 + +from django.db import migrations, models +import myauth.models + + +class Migration(migrations.Migration): + + dependencies = [ + ('myauth', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='user', + name='username', + field=models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and ./+/-/_ only.', max_length=150, unique=True, validators=[myauth.models.UnicodeUsernameValidator()], verbose_name='username'), + ), + ] diff --git a/myauth/models.py b/myauth/models.py index 3d30525..4afc27c 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -1,5 +1,30 @@ from django.contrib.auth.models import AbstractUser +from django.core import validators +from django.db import models +from django.utils.deconstruct import deconstructible +from django.utils.translation import gettext_lazy as _ + + +@deconstructible +class UnicodeUsernameValidator(validators.RegexValidator): + regex = r'^[\w.+-]+\Z' + message = _( + 'Enter a valid username. This value may contain only letters, ' + 'numbers, and ./+/-/_ characters.' + ) + flags = 0 class User(AbstractUser): - pass + username_validator = UnicodeUsernameValidator() + + username = models.CharField( + _('username'), + max_length=150, + unique=True, + help_text=_('Required. 150 characters or fewer. Letters, digits and ./+/-/_ only.'), + validators=[username_validator], + error_messages={ + 'unique': _("A user with that username already exists."), + }, + ) From f438d0e947a777ca949b904d7b3793b629f2d59d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 15 May 2020 12:44:10 +0300 Subject: [PATCH 125/511] Trim salt when creating the challenge. --- django_etesync/views.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index eaf0e35..1a7c5bc 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -309,7 +309,8 @@ class AuthenticationViewSet(viewsets.ViewSet): def get_encryption_key(self, salt): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) - return nacl.hash.blake2b(b'', key=key, salt=salt, person=b'etesync-auth', encoder=nacl.encoding.RawEncoder) + return nacl.hash.blake2b(b'', key=key, salt=salt[:nacl.hash.BLAKE2B_SALTBYTES], person=b'etesync-auth', + encoder=nacl.encoding.RawEncoder) def get_queryset(self): return User.objects.all() From 48ebbfb3223b6fc28eb00ce0c63cf1e58406b4cd Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 15 May 2020 12:51:05 +0300 Subject: [PATCH 126/511] Disable host verification for debug mode. Was causing issues with mitm proxy and etc which was a pain. --- django_etesync/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 1a7c5bc..894f06d 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -403,7 +403,7 @@ class AuthenticationViewSet(viewsets.ViewSet): elif challenge_data['userId'] != user.id: content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif host != request.get_host(): + elif not settings.DEBUG and host != request.get_host(): detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) content = {'code': 'wrong_host', 'detail': detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) From 644539bd6834f0c2934b21f644a1394c3fabc2d0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 15 May 2020 12:44:25 +0300 Subject: [PATCH 127/511] Reset view: adjust reset view path and class. --- django_etesync/views.py | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 894f06d..dd110db 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -422,16 +422,22 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_400_BAD_REQUEST) -class ResetViewSet(BaseViewSet): +class TestAuthenticationViewSet(viewsets.ViewSet): + authentication_classes = BaseViewSet.authentication_classes + permission_classes = BaseViewSet.permission_classes allowed_methods = ['POST'] - def post(self, request, *args, **kwargs): + def list(self, request): + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + @action_decorator(detail=False, methods=['POST']) + def reset(self, request, *args, **kwargs): # Only run when in DEBUG mode! It's only used for tests if not settings.DEBUG: return HttpResponseBadRequest("Only allowed in debug mode.") # Only allow local users, for extra safety - if not getattr(request.user, User.USERNAME_FIELD).endswith('@localhost'): + if not getattr(request.user, User.EMAIL_FIELD).endswith('@localhost'): return HttpResponseBadRequest("Endpoint not allowed for user.") # Delete all of the journal data for this user for a clear test env @@ -440,6 +446,3 @@ class ResetViewSet(BaseViewSet): # FIXME: also delete chunk files!!! return HttpResponse() - - -reset = ResetViewSet.as_view({'post': 'post'}) From bced00dc8aa1df3ecb1c734e91f6249ea65b762d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 15 May 2020 13:03:04 +0300 Subject: [PATCH 128/511] Enable logout for now so client tests pass. --- django_etesync/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index dd110db..db8550b 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -419,7 +419,7 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST']) def logout(self, request): # FIXME: expire the token - we need better token handling - using knox? Something else? - return Response({}, status=status.HTTP_400_BAD_REQUEST) + return Response({}, status=status.HTTP_200_OK) class TestAuthenticationViewSet(viewsets.ViewSet): From bd1d11fe5f81d8e2d6f2b0bc55d3e17d291154f6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 18 May 2020 16:13:48 +0300 Subject: [PATCH 129/511] Fix signup and let signup to an empty account. --- django_etesync/serializers.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 7173d73..88d237f 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -219,12 +219,17 @@ class AuthenticationSignupSerializer(serializers.Serializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" + user_data = validated_data.pop('user') salt = validated_data.pop('salt') pubkey = validated_data.pop('pubkey') with transaction.atomic(): - instance = User.objects.create(**validated_data) + instance = User.objects.get_or_create(**user_data) + if hasattr(instance, 'userinfo'): + raise serializers.ValidationError('User already exists') + instance.set_unusable_password() + # FIXME: send email verification models.UserInfo.objects.create(salt=salt, pubkey=pubkey, owner=instance) From 00a80740caf29dbede57e5c1bd88b59ef69736d4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 10:48:39 +0300 Subject: [PATCH 130/511] Collection/item create/update require stoken. --- django_etesync/serializers.py | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 88d237f..a5755d5 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -112,18 +112,23 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): class CollectionItemSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() + stoken = serializers.CharField(allow_null=True) content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.CollectionItem - fields = ('uid', 'version', 'encryptionKey', 'content') + fields = ('uid', 'version', 'encryptionKey', 'content', 'stoken') def create(self, validated_data): """Function that's called when this serializer creates an item""" + stoken = validated_data.pop('stoken') revision_data = validated_data.pop('content') instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): + if stoken is not None: + raise serializers.ValidationError('Stoken is not None') + instance.save() process_revisions_for_item(instance, revision_data) @@ -132,9 +137,13 @@ class CollectionItemSerializer(serializers.ModelSerializer): def update(self, instance, validated_data): """Function that's called when this serializer is meant to update an item""" + stoken = validated_data.pop('stoken') revision_data = validated_data.pop('content') with transaction.atomic(): + if stoken != instance.stoken: + raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(instance.stoken, stoken)) + # We don't have to use select_for_update here because the unique constraint on current guards against # the race condition. But it's a good idea because it'll lock and wait rather than fail. current_revision = instance.revisions.filter(current=True).select_for_update().first() @@ -149,7 +158,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') - stoken = serializers.CharField(read_only=True) + stoken = serializers.CharField(allow_null=True) content = CollectionItemRevisionSerializer(many=False) class Meta: @@ -164,11 +173,15 @@ class CollectionSerializer(serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" + stoken = validated_data.pop('stoken') revision_data = validated_data.pop('content') encryption_key = validated_data.pop('encryptionKey') instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): + if stoken is not None: + raise serializers.ValidationError('Stoken is not None') + instance.save() main_item = models.CollectionItem.objects.create( uid=None, encryptionKey=None, version=instance.version, collection=instance) @@ -185,9 +198,13 @@ class CollectionSerializer(serializers.ModelSerializer): def update(self, instance, validated_data): """Function that's called when this serializer is meant to update an item""" + stoken = validated_data.pop('stoken') revision_data = validated_data.pop('content') with transaction.atomic(): + if stoken != instance.stoken: + raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(instance.stoken, stoken)) + main_item = instance.main_item # We don't have to use select_for_update here because the unique constraint on current guards against # the race condition. But it's a good idea because it'll lock and wait rather than fail. From 775f438e611d36ea81231500f6c593b67dcb2dc3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 11:20:02 +0300 Subject: [PATCH 131/511] Change deps to be pairs of uid/stoken. --- django_etesync/serializers.py | 17 +++++++++++++++++ django_etesync/views.py | 5 +++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index a5755d5..71ddcb3 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -155,6 +155,23 @@ class CollectionItemSerializer(serializers.ModelSerializer): return instance +class CollectionItemDepSerializer(serializers.ModelSerializer): + stoken = serializers.CharField() + + class Meta: + model = models.CollectionItem + fields = ('uid', 'stoken') + + def validate(self, data): + for item_data in data: + item = self.__class__.Meta.model.objects.get(uid=item_data['uid']) + stoken = item_data['stoken'] + if item.stoken != stoken: + raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(item.stoken, stoken)) + + return data + + class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') diff --git a/django_etesync/views.py b/django_etesync/views.py index db8550b..28dc601 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -43,6 +43,7 @@ from .serializers import ( AuthenticationLoginInnerSerializer, CollectionSerializer, CollectionItemSerializer, + CollectionItemDepSerializer, CollectionItemRevisionSerializer, CollectionItemChunkSerializer, UserSerializer, @@ -234,15 +235,15 @@ class CollectionItemViewSet(BaseViewSet): collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) items = request.data.get('items') - # FIXME: deps should actually be just pairs of uid and stoken deps = request.data.get('deps', None) serializer = self.get_serializer_class()(data=items, context=self.get_serializer_context(), many=True) - deps_serializer = self.get_serializer_class()(data=deps, context=self.get_serializer_context(), many=True) + deps_serializer = CollectionItemDepSerializer(data=deps, context=self.get_serializer_context(), many=True) if serializer.is_valid() and (deps is None or deps_serializer.is_valid()): try: with transaction.atomic(): collections = serializer.save(collection=collection_object) except IntegrityError: + # FIXME: should return the items with a bad token (including deps) so we don't have to fetch them after content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) From 306e7dcd113477a65220db20e2e0ca92afef673a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 11:44:20 +0300 Subject: [PATCH 132/511] Item deps: fix. --- django_etesync/serializers.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 71ddcb3..3f7c28c 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -163,11 +163,10 @@ class CollectionItemDepSerializer(serializers.ModelSerializer): fields = ('uid', 'stoken') def validate(self, data): - for item_data in data: - item = self.__class__.Meta.model.objects.get(uid=item_data['uid']) - stoken = item_data['stoken'] - if item.stoken != stoken: - raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(item.stoken, stoken)) + item = self.__class__.Meta.model.objects.get(uid=data['uid']) + stoken = data['stoken'] + if item.stoken != stoken: + raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(item.stoken, stoken)) return data From 23dcbc1f9ea60c689e6a27acb31ba7657cdfa07a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 12:57:18 +0300 Subject: [PATCH 133/511] CollectionItem: always run both serializers when serializing. --- django_etesync/views.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 28dc601..034eaed 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -236,9 +236,13 @@ class CollectionItemViewSet(BaseViewSet): items = request.data.get('items') deps = request.data.get('deps', None) + # FIXME: It should just be one serializer serializer = self.get_serializer_class()(data=items, context=self.get_serializer_context(), many=True) deps_serializer = CollectionItemDepSerializer(data=deps, context=self.get_serializer_context(), many=True) - if serializer.is_valid() and (deps is None or deps_serializer.is_valid()): + + ser_valid = serializer.is_valid() + deps_ser_valid = (deps is None or deps_serializer.is_valid()) + if ser_valid and deps_ser_valid: try: with transaction.atomic(): collections = serializer.save(collection=collection_object) From 4c7e30eca579d32eb4092b09c760483eb0d6ca3a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 13:00:54 +0300 Subject: [PATCH 134/511] CollectionItem: implement both update and create. --- django_etesync/serializers.py | 37 +++++++++++++++-------------------- django_etesync/views.py | 4 +++- 2 files changed, 19 insertions(+), 22 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 3f7c28c..5fd6ea8 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -123,36 +123,31 @@ class CollectionItemSerializer(serializers.ModelSerializer): """Function that's called when this serializer creates an item""" stoken = validated_data.pop('stoken') revision_data = validated_data.pop('content') - instance = self.__class__.Meta.model(**validated_data) + uid = validated_data.pop('uid') + + Model = self.__class__.Meta.model with transaction.atomic(): - if stoken is not None: - raise serializers.ValidationError('Stoken is not None') + instance, created = Model.objects.get_or_create(uid=uid, defaults=validated_data) + cur_stoken = instance.stoken if not created else None - instance.save() + if cur_stoken != stoken: + raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(cur_stoken, stoken)) + + if not created: + # We don't have to use select_for_update here because the unique constraint on current guards against + # the race condition. But it's a good idea because it'll lock and wait rather than fail. + current_revision = instance.revisions.filter(current=True).select_for_update().first() + current_revision.current = None + current_revision.save() process_revisions_for_item(instance, revision_data) return instance def update(self, instance, validated_data): - """Function that's called when this serializer is meant to update an item""" - stoken = validated_data.pop('stoken') - revision_data = validated_data.pop('content') - - with transaction.atomic(): - if stoken != instance.stoken: - raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(instance.stoken, stoken)) - - # We don't have to use select_for_update here because the unique constraint on current guards against - # the race condition. But it's a good idea because it'll lock and wait rather than fail. - current_revision = instance.revisions.filter(current=True).select_for_update().first() - current_revision.current = None - current_revision.save() - - process_revisions_for_item(instance, revision_data) - - return instance + # We never update, we always update in the create method + raise NotImplementedError() class CollectionItemDepSerializer(serializers.ModelSerializer): diff --git a/django_etesync/views.py b/django_etesync/views.py index 034eaed..24b62fa 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -187,8 +187,10 @@ class CollectionItemViewSet(BaseViewSet): # We can't have destroy because we need to get data from the user (in the body) such as hmac. return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + def update(self, request, collection_uid=None, uid=None): + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + def partial_update(self, request, collection_uid=None, uid=None): - # FIXME: implement, or should it be implemented elsewhere? return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def list(self, request, collection_uid=None): From f7c66eaadb0842386548ee6a9910e2e228094931 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 13:10:50 +0300 Subject: [PATCH 135/511] CollectionItem: add a batch endpoint for batch operations. --- django_etesync/views.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 24b62fa..c905a54 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -232,6 +232,11 @@ class CollectionItemViewSet(BaseViewSet): } return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + @action_decorator(detail=False, methods=['POST']) + def batch(self, request, collection_uid=None): + # FIXME: different to transaction slightly + return self.transaction(request, collection_uid) + @action_decorator(detail=False, methods=['POST']) def transaction(self, request, collection_uid=None): collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) From eeaea6e6aba90e54fa84bb4b5fcc27e43e526cf4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 13:19:25 +0300 Subject: [PATCH 136/511] Transaction: return 200 rather than 201. --- django_etesync/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index c905a54..c56878b 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -261,7 +261,7 @@ class CollectionItemViewSet(BaseViewSet): ret = { "data": [collection.stoken for collection in collections], } - return Response(ret, status=status.HTTP_201_CREATED) + return Response(ret, status=status.HTTP_200_OK) return Response( { From ae4aafcf96f955464c32c6c9fb418f8668937cb3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 14:54:44 +0300 Subject: [PATCH 137/511] Transaction: make it possible to pass a global stoken to block by. --- django_etesync/views.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index c56878b..ad57d28 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -70,16 +70,27 @@ class BaseViewSet(viewsets.ModelViewSet): user = self.request.user return queryset.filter(members__user=user) - def filter_by_stoken_and_limit(self, request, queryset): + def get_stoken_rev(self, request): stoken = request.GET.get('stoken', None) + + if stoken is not None: + return get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + + return None + + def filter_by_stoken_and_limit(self, request, queryset): limit = int(request.GET.get('limit', 50)) stoken_id_field = self.stoken_id_field + '__id' - if stoken is not None: - last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + stoken_rev = self.get_stoken_rev(request) + if stoken_rev is not None: + last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken_rev.uid) filter_by = {stoken_id_field + '__gt': last_rev.id} queryset = queryset.filter(**filter_by) + stoken = stoken_rev.uid + else: + stoken = None new_stoken_id = queryset.aggregate(stoken_id=Max(stoken_id_field))['stoken_id'] new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken @@ -239,8 +250,13 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def transaction(self, request, collection_uid=None): + stoken = request.GET.get('stoken', None) collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + if stoken is not None and stoken != collection_object.stoken: + content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + items = request.data.get('items') deps = request.data.get('deps', None) # FIXME: It should just be one serializer From e851fb98776ddb3025111cbaa315d64c025fbb2b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 15:28:20 +0300 Subject: [PATCH 138/511] Views: fix wrong items name. --- django_etesync/views.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index ad57d28..9d5d938 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -268,14 +268,14 @@ class CollectionItemViewSet(BaseViewSet): if ser_valid and deps_ser_valid: try: with transaction.atomic(): - collections = serializer.save(collection=collection_object) + items = serializer.save(collection=collection_object) except IntegrityError: # FIXME: should return the items with a bad token (including deps) so we don't have to fetch them after content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) ret = { - "data": [collection.stoken for collection in collections], + "data": [item.stoken for item in items], } return Response(ret, status=status.HTTP_200_OK) From b6571c93f63d6ebb48194b66ff82dda34ca8e9dc Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 15:33:10 +0300 Subject: [PATCH 139/511] Collection: fix stoken and add cstoken for the collection token. --- django_etesync/models.py | 4 +++ django_etesync/serializers.py | 3 +- django_etesync/views.py | 54 +++++++++++++++++------------------ 3 files changed, 33 insertions(+), 28 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 0d0f3dc..29c7e57 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -46,6 +46,10 @@ class Collection(models.Model): @cached_property def stoken(self): + return self.main_item.stoken + + @cached_property + def cstoken(self): last_revision = CollectionItemRevision.objects.filter(item__collection=self).last() if last_revision is None: # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 5fd6ea8..e5a9e4f 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -169,12 +169,13 @@ class CollectionItemDepSerializer(serializers.ModelSerializer): class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') + cstoken = serializers.CharField(read_only=True) stoken = serializers.CharField(allow_null=True) content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.Collection - fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'stoken') + fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'cstoken', 'stoken') def get_access_level_from_context(self, obj): request = self.context.get('request', None) diff --git a/django_etesync/views.py b/django_etesync/views.py index 9d5d938..081bd37 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -56,7 +56,7 @@ User = get_user_model() class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) - stoken_id_field = None + cstoken_id_field = None def get_serializer_class(self): serializer_class = self.serializer_class @@ -70,32 +70,32 @@ class BaseViewSet(viewsets.ModelViewSet): user = self.request.user return queryset.filter(members__user=user) - def get_stoken_rev(self, request): - stoken = request.GET.get('stoken', None) + def get_cstoken_rev(self, request): + cstoken = request.GET.get('cstoken', None) - if stoken is not None: - return get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken) + if cstoken is not None: + return get_object_or_404(CollectionItemRevision.objects.all(), uid=cstoken) return None - def filter_by_stoken_and_limit(self, request, queryset): + def filter_by_cstoken_and_limit(self, request, queryset): limit = int(request.GET.get('limit', 50)) - stoken_id_field = self.stoken_id_field + '__id' + cstoken_id_field = self.cstoken_id_field + '__id' - stoken_rev = self.get_stoken_rev(request) - if stoken_rev is not None: - last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=stoken_rev.uid) - filter_by = {stoken_id_field + '__gt': last_rev.id} + cstoken_rev = self.get_cstoken_rev(request) + if cstoken_rev is not None: + last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=cstoken_rev.uid) + filter_by = {cstoken_id_field + '__gt': last_rev.id} queryset = queryset.filter(**filter_by) - stoken = stoken_rev.uid + cstoken = cstoken_rev.uid else: - stoken = None + cstoken = None - new_stoken_id = queryset.aggregate(stoken_id=Max(stoken_id_field))['stoken_id'] - new_stoken = CollectionItemRevision.objects.get(id=new_stoken_id).uid if new_stoken_id is not None else stoken + new_cstoken_id = queryset.aggregate(cstoken_id=Max(cstoken_id_field))['cstoken_id'] + new_cstoken = CollectionItemRevision.objects.get(id=new_cstoken_id).uid if new_cstoken_id is not None else cstoken - return queryset[:limit], new_stoken + return queryset[:limit], new_cstoken class CollectionViewSet(BaseViewSet): @@ -104,7 +104,7 @@ class CollectionViewSet(BaseViewSet): queryset = Collection.objects.all() serializer_class = CollectionSerializer lookup_field = 'uid' - stoken_id_field = 'items__revisions' + cstoken_id_field = 'items__revisions' def get_queryset(self, queryset=None): if queryset is None: @@ -139,14 +139,14 @@ class CollectionViewSet(BaseViewSet): def list(self, request): queryset = self.get_queryset() - queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_cstoken = self.filter_by_cstoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) ret = { 'data': serializer.data, } - return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + return Response(ret, headers={'X-EteSync-SToken': new_cstoken}) class CollectionItemViewSet(BaseViewSet): @@ -155,7 +155,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = 'uid' - stoken_id_field = 'revisions' + cstoken_id_field = 'revisions' def get_queryset(self): collection_uid = self.kwargs['collection_uid'] @@ -206,14 +206,14 @@ class CollectionItemViewSet(BaseViewSet): def list(self, request, collection_uid=None): queryset = self.get_queryset() - queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_cstoken = self.filter_by_cstoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) ret = { 'data': serializer.data, } - return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + return Response(ret, headers={'X-EteSync-SToken': new_cstoken}) @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): @@ -234,14 +234,14 @@ class CollectionItemViewSet(BaseViewSet): if isinstance(request.data, list): queryset = queryset.filter(uid__in=request.data) - queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_cstoken = self.filter_by_cstoken_and_limit(request, queryset) serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) ret = { 'data': serializer.data, } - return Response(ret, headers={'X-EteSync-SToken': new_stoken}) + return Response(ret, headers={'X-EteSync-SToken': new_cstoken}) @action_decorator(detail=False, methods=['POST']) def batch(self, request, collection_uid=None): @@ -250,11 +250,11 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def transaction(self, request, collection_uid=None): - stoken = request.GET.get('stoken', None) + cstoken = request.GET.get('cstoken', None) collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - if stoken is not None and stoken != collection_object.stoken: - content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} + if cstoken is not None and cstoken != collection_object.cstoken: + content = {'code': 'stale_cstoken', 'detail': 'CSToken is too old'} return Response(content, status=status.HTTP_400_BAD_REQUEST) items = request.data.get('items') From c63210fe77ff0c3d839174e32b4fe8f91818f590 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 16:16:40 +0300 Subject: [PATCH 140/511] CollectionItem: implement batch updating. --- django_etesync/serializers.py | 3 ++- django_etesync/views.py | 38 +++++++++++++++++++++++++++++++---- 2 files changed, 36 insertions(+), 5 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index e5a9e4f..773a8a2 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -121,6 +121,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" + validate_stoken = self.context.get('validate_stoken', False) stoken = validated_data.pop('stoken') revision_data = validated_data.pop('content') uid = validated_data.pop('uid') @@ -131,7 +132,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): instance, created = Model.objects.get_or_create(uid=uid, defaults=validated_data) cur_stoken = instance.stoken if not created else None - if cur_stoken != stoken: + if validate_stoken and cur_stoken != stoken: raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(cur_stoken, stoken)) if not created: diff --git a/django_etesync/views.py b/django_etesync/views.py index 081bd37..ab5c0c7 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -245,8 +245,36 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def batch(self, request, collection_uid=None): - # FIXME: different to transaction slightly - return self.transaction(request, collection_uid) + cstoken = request.GET.get('cstoken', None) + collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + + if cstoken is not None and cstoken != collection_object.cstoken: + content = {'code': 'stale_cstoken', 'detail': 'CSToken is too old'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + items = request.data.get('items') + context = self.get_serializer_context() + serializer = self.get_serializer_class()(data=items, context=context, many=True) + + if serializer.is_valid(): + try: + with transaction.atomic(): + items = serializer.save(collection=collection_object) + except IntegrityError: + # FIXME: should return the items with a bad token (including deps) so we don't have to fetch them after + content = {'code': 'integrity_error'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + ret = { + "data": [item.stoken for item in items], + } + return Response(ret, status=status.HTTP_200_OK) + + return Response( + { + "items": serializer.errors, + }, + status=status.HTTP_400_BAD_REQUEST) @action_decorator(detail=False, methods=['POST']) def transaction(self, request, collection_uid=None): @@ -260,8 +288,10 @@ class CollectionItemViewSet(BaseViewSet): items = request.data.get('items') deps = request.data.get('deps', None) # FIXME: It should just be one serializer - serializer = self.get_serializer_class()(data=items, context=self.get_serializer_context(), many=True) - deps_serializer = CollectionItemDepSerializer(data=deps, context=self.get_serializer_context(), many=True) + context = self.get_serializer_context() + context.update({'validate_stoken': True}) + serializer = self.get_serializer_class()(data=items, context=context, many=True) + deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) ser_valid = serializer.is_valid() deps_ser_valid = (deps is None or deps_serializer.is_valid()) From 9bbb7ef3d75a88ab294187341d7febdd7c4c1fd8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 17:29:54 +0300 Subject: [PATCH 141/511] Fix filter by cstoken function to not fetch twice. --- django_etesync/views.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index ab5c0c7..aa3c148 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -85,8 +85,7 @@ class BaseViewSet(viewsets.ModelViewSet): cstoken_rev = self.get_cstoken_rev(request) if cstoken_rev is not None: - last_rev = get_object_or_404(CollectionItemRevision.objects.all(), uid=cstoken_rev.uid) - filter_by = {cstoken_id_field + '__gt': last_rev.id} + filter_by = {cstoken_id_field + '__gt': cstoken_rev.id} queryset = queryset.filter(**filter_by) cstoken = cstoken_rev.uid else: From aaee8f5e381f688dd37c8bccc467a37917ad1dc0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 17:41:27 +0300 Subject: [PATCH 142/511] Fix new_cstoken getting for list functions. We were getting the general cstoken, and were not honouring our limit. --- django_etesync/views.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index aa3c148..5a4101c 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -91,10 +91,11 @@ class BaseViewSet(viewsets.ModelViewSet): else: cstoken = None + queryset = queryset[:limit] new_cstoken_id = queryset.aggregate(cstoken_id=Max(cstoken_id_field))['cstoken_id'] new_cstoken = CollectionItemRevision.objects.get(id=new_cstoken_id).uid if new_cstoken_id is not None else cstoken - return queryset[:limit], new_cstoken + return queryset, new_cstoken class CollectionViewSet(BaseViewSet): From c30cc2f229b5e79f7b8d8e7684c8d52a8115f3bb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 19 May 2020 17:57:51 +0300 Subject: [PATCH 143/511] Improve and rename bulk_get to filter by item stokens too + cstoken changes Also change how we return cstokens --- django_etesync/serializers.py | 8 ++++ django_etesync/views.py | 71 +++++++++++++++++++++++++---------- 2 files changed, 60 insertions(+), 19 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 773a8a2..6413ab1 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -167,6 +167,14 @@ class CollectionItemDepSerializer(serializers.ModelSerializer): return data +class CollectionItemBulkGetSerializer(serializers.ModelSerializer): + stoken = serializers.CharField(required=False) + + class Meta: + model = models.CollectionItem + fields = ('uid', 'stoken') + + class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') diff --git a/django_etesync/views.py b/django_etesync/views.py index 5a4101c..d100d7c 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -43,6 +43,7 @@ from .serializers import ( AuthenticationLoginInnerSerializer, CollectionSerializer, CollectionItemSerializer, + CollectionItemBulkGetSerializer, CollectionItemDepSerializer, CollectionItemRevisionSerializer, CollectionItemChunkSerializer, @@ -78,22 +79,33 @@ class BaseViewSet(viewsets.ModelViewSet): return None - def filter_by_cstoken_and_limit(self, request, queryset): - limit = int(request.GET.get('limit', 50)) - + def filter_by_cstoken(self, request, queryset): cstoken_id_field = self.cstoken_id_field + '__id' cstoken_rev = self.get_cstoken_rev(request) if cstoken_rev is not None: filter_by = {cstoken_id_field + '__gt': cstoken_rev.id} queryset = queryset.filter(**filter_by) - cstoken = cstoken_rev.uid - else: - cstoken = None - queryset = queryset[:limit] + return queryset, cstoken_rev + + def get_queryset_cstoken(self, queryset): + cstoken_id_field = self.cstoken_id_field + '__id' + new_cstoken_id = queryset.aggregate(cstoken_id=Max(cstoken_id_field))['cstoken_id'] - new_cstoken = CollectionItemRevision.objects.get(id=new_cstoken_id).uid if new_cstoken_id is not None else cstoken + new_cstoken = new_cstoken_id and CollectionItemRevision.objects.get(id=new_cstoken_id).uid + + return queryset, new_cstoken + + def filter_by_cstoken_and_limit(self, request, queryset): + limit = int(request.GET.get('limit', 50)) + + queryset, cstoken_rev = self.filter_by_cstoken(request, queryset) + cstoken = cstoken_rev.uid if cstoken_rev is not None else None + + queryset = queryset[:limit] + queryset, new_cstoken = self.get_queryset_cstoken(queryset) + new_cstoken = new_cstoken or cstoken return queryset, new_cstoken @@ -145,8 +157,9 @@ class CollectionViewSet(BaseViewSet): ret = { 'data': serializer.data, + 'cstoken': new_cstoken, } - return Response(ret, headers={'X-EteSync-SToken': new_cstoken}) + return Response(ret) class CollectionItemViewSet(BaseViewSet): @@ -212,8 +225,9 @@ class CollectionItemViewSet(BaseViewSet): ret = { 'data': serializer.data, + 'cstoken': new_cstoken, } - return Response(ret, headers={'X-EteSync-SToken': new_cstoken}) + return Response(ret) @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): @@ -227,21 +241,40 @@ class CollectionItemViewSet(BaseViewSet): } return Response(ret) + # FIXME: rename to something consistent with what the clients have - maybe list_updates? @action_decorator(detail=False, methods=['POST']) - def bulk_get(self, request, collection_uid=None): + def fetch_updates(self, request, collection_uid=None): queryset = self.get_queryset() - if isinstance(request.data, list): - queryset = queryset.filter(uid__in=request.data) + serializer = CollectionItemBulkGetSerializer(data=request.data, many=True) + if serializer.is_valid(): + # FIXME: make configurable? + item_limit = 200 - queryset, new_cstoken = self.filter_by_cstoken_and_limit(request, queryset) + if len(serializer.validated_data) > item_limit: + content = {'code': 'too_many_items', + 'detail': 'Request has too many items. Limit: {}'. format(item_limit)} + return Response(content, status=status.HTTP_400_BAD_REQUEST) - serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) + queryset, cstoken_rev = self.filter_by_cstoken(request, queryset) - ret = { - 'data': serializer.data, - } - return Response(ret, headers={'X-EteSync-SToken': new_cstoken}) + uids, stokens = zip(*[(item['uid'], item.get('stoken')) for item in serializer.validated_data]) + rev_ids = CollectionItemRevision.objects.filter(uid__in=stokens, current=True).values_list('id', flat=True) + queryset = queryset.filter(uid__in=uids).exclude(revisions__id__in=rev_ids) + + queryset, new_cstoken = self.get_queryset_cstoken(queryset) + cstoken = cstoken_rev and cstoken_rev.uid + new_cstoken = new_cstoken or cstoken + + serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) + + ret = { + 'data': serializer.data, + 'cstoken': new_cstoken, + } + return Response(ret) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) @action_decorator(detail=False, methods=['POST']) def batch(self, request, collection_uid=None): From 4ca74bc69ba6616d5edf99ef30d36a66c6b07010 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 20 May 2020 13:47:06 +0300 Subject: [PATCH 144/511] Permissions: start from scratch and add IsCollectionAdmin permission. --- django_etesync/permissions.py | 53 ++++++++--------------------------- 1 file changed, 11 insertions(+), 42 deletions(-) diff --git a/django_etesync/permissions.py b/django_etesync/permissions.py index f553930..29806c6 100644 --- a/django_etesync/permissions.py +++ b/django_etesync/permissions.py @@ -13,53 +13,22 @@ # along with this program. If not, see . from rest_framework import permissions -from journal.models import Journal, JournalMember +from django_etesync.models import Collection, AccessLevels -class IsOwnerOrReadOnly(permissions.BasePermission): +class IsCollectionAdmin(permissions.BasePermission): """ - Custom permission to only allow owners of an object to edit it. - """ - - def has_object_permission(self, request, view, obj): - if request.method in permissions.SAFE_METHODS: - return True - - return obj.owner == request.user - - -class IsJournalOwner(permissions.BasePermission): - """ - Custom permission to only allow owners of a journal to view it - """ - - def has_permission(self, request, view): - journal_uid = view.kwargs['journal_uid'] - try: - journal = view.get_journal_queryset().get(uid=journal_uid) - return journal.owner == request.user - except Journal.DoesNotExist: - # If the journal does not exist, we want to 404 later, not permission denied. - return True - - -class IsMemberReadOnly(permissions.BasePermission): - """ - Custom permission to make a journal read only if a read only member + Custom permission to only allow owners of a collection to view it """ + message = 'Only collection admins can perform this operation.' + code = 'admin_access_required' def has_permission(self, request, view): - if request.method in permissions.SAFE_METHODS: - return True - - journal_uid = view.kwargs['journal_uid'] + collection_uid = view.kwargs['collection_uid'] try: - journal = view.get_journal_queryset().get(uid=journal_uid) - member = journal.members.get(user=request.user) - return not member.readOnly - except Journal.DoesNotExist: - # If the journal does not exist, we want to 404 later, not permission denied. - return True - except JournalMember.DoesNotExist: - # Not being a member means we are the owner. + collection = view.get_collection_queryset().get(uid=collection_uid) + member = collection.members.filter(user=request.user).first() + return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) + except Collection.DoesNotExist: + # If the collection does not exist, we want to 404 later, not permission denied. return True From edd88427b0fc739a8610f1e6135680f1627f7c47 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 20 May 2020 13:48:46 +0300 Subject: [PATCH 145/511] Add a viewset to control collection membership. --- django_etesync/models.py | 11 ++++++----- django_etesync/serializers.py | 26 +++++++++++++++++++++++++- django_etesync/views.py | 32 ++++++++++++++++++++++++++++++-- 3 files changed, 61 insertions(+), 8 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 29c7e57..36851a1 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -121,12 +121,13 @@ class RevisionChunkRelation(models.Model): ordering = ('id', ) -class CollectionMember(models.Model): - class AccessLevels(models.TextChoices): - ADMIN = 'adm' - READ_WRITE = 'rw' - READ_ONLY = 'ro' +class AccessLevels(models.TextChoices): + ADMIN = 'adm' + READ_WRITE = 'rw' + READ_ONLY = 'ro' + +class CollectionMember(models.Model): collection = models.ForeignKey(Collection, related_name='members', on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 6413ab1..06bb7a9 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -211,7 +211,7 @@ class CollectionSerializer(serializers.ModelSerializer): models.CollectionMember(collection=instance, user=validated_data.get('owner'), - accessLevel=models.CollectionMember.AccessLevels.ADMIN, + accessLevel=models.AccessLevels.ADMIN, encryptionKey=encryption_key, ).save() @@ -238,6 +238,30 @@ class CollectionSerializer(serializers.ModelSerializer): return instance +class CollectionMemberSerializer(serializers.ModelSerializer): + username = serializers.SlugRelatedField( + source='user', + slug_field=User.USERNAME_FIELD, + queryset=User.objects + ) + encryptionKey = BinaryBase64Field() + + class Meta: + model = models.CollectionMember + fields = ('username', 'encryptionKey', 'accessLevel') + + def create(self, validated_data): + raise NotImplementedError() + + def update(self, instance, validated_data): + with transaction.atomic(): + # We only allow updating accessLevel + instance.accessLevel = validated_data.pop('accessLevel') + instance.save() + + return instance + + class UserSerializer(serializers.ModelSerializer): class Meta: model = User diff --git a/django_etesync/views.py b/django_etesync/views.py index d100d7c..5703410 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -33,8 +33,8 @@ import nacl.signing import nacl.secret import nacl.hash -from . import app_settings -from .models import Collection, CollectionItem, CollectionItemRevision +from . import app_settings, permissions +from .models import Collection, CollectionItem, CollectionItemRevision, CollectionMember from .serializers import ( b64encode, AuthenticationSignupSerializer, @@ -47,6 +47,7 @@ from .serializers import ( CollectionItemDepSerializer, CollectionItemRevisionSerializer, CollectionItemChunkSerializer, + CollectionMemberSerializer, UserSerializer, ) @@ -395,6 +396,33 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return serve(request, basename, dirname) +class CollectionMemberViewSet(BaseViewSet): + allowed_methods = ['GET', 'PUT', 'DELETE'] + permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdmin, ) + queryset = CollectionMember.objects.all() + serializer_class = CollectionMemberSerializer + lookup_field = 'user__' + User.USERNAME_FIELD + lookup_url_kwarg = 'username' + + # FIXME: need to make sure that there's always an admin, and maybe also don't let an owner remove adm access + # (if we want to transfer, we need to do that specifically) + + def get_queryset(self, queryset=None): + collection_uid = self.kwargs['collection_uid'] + try: + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + except Collection.DoesNotExist: + raise Http404('Collection does not exist') + + if queryset is None: + queryset = type(self).queryset + + return queryset.filter(collection=collection) + + def create(self, request): + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + + class AuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] From 8d1c02dcb99944e38dd69e79f2ae088871811990 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 20 May 2020 14:30:09 +0300 Subject: [PATCH 146/511] Collection invitation: implement creating and manipulating collections invitations. --- .../migrations/0003_collectioninvitation.py | 31 +++++++++++++++ .../0004_collectioninvitation_version.py | 18 +++++++++ django_etesync/models.py | 22 +++++++++++ django_etesync/serializers.py | 39 +++++++++++++++++++ django_etesync/views.py | 36 ++++++++++++++++- 5 files changed, 145 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0003_collectioninvitation.py create mode 100644 django_etesync/migrations/0004_collectioninvitation_version.py diff --git a/django_etesync/migrations/0003_collectioninvitation.py b/django_etesync/migrations/0003_collectioninvitation.py new file mode 100644 index 0000000..3880a63 --- /dev/null +++ b/django_etesync/migrations/0003_collectioninvitation.py @@ -0,0 +1,31 @@ +# Generated by Django 3.0.3 on 2020-05-20 11:03 + +from django.conf import settings +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ('django_etesync', '0002_userinfo'), + ] + + operations = [ + migrations.CreateModel( + name='CollectionInvitation', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), + ('signedEncryptionKey', models.BinaryField()), + ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), + ('fromMember', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etesync.CollectionMember')), + ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='incoming_invitations', to=settings.AUTH_USER_MODEL)), + ], + options={ + 'unique_together': {('user', 'fromMember')}, + }, + ), + ] diff --git a/django_etesync/migrations/0004_collectioninvitation_version.py b/django_etesync/migrations/0004_collectioninvitation_version.py new file mode 100644 index 0000000..3fbaed9 --- /dev/null +++ b/django_etesync/migrations/0004_collectioninvitation_version.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-05-21 14:45 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0003_collectioninvitation'), + ] + + operations = [ + migrations.AddField( + model_name='collectioninvitation', + name='version', + field=models.PositiveSmallIntegerField(default=1), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 36851a1..e0a79f6 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -144,6 +144,28 @@ class CollectionMember(models.Model): return '{} {}'.format(self.collection.uid, self.user) +class CollectionInvitation(models.Model): + uid = models.CharField(db_index=True, blank=False, null=False, + max_length=44, validators=[Base64Url256BitValidator]) + version = models.PositiveSmallIntegerField(default=1) + fromMember = models.ForeignKey(CollectionMember, on_delete=models.CASCADE) + # FIXME: make sure to delete all invitations for the same collection once one is accepted + + user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name='incoming_invitations', on_delete=models.CASCADE) + signedEncryptionKey = models.BinaryField(editable=False, blank=False, null=False) + accessLevel = models.CharField( + max_length=3, + choices=AccessLevels.choices, + default=AccessLevels.READ_ONLY, + ) + + class Meta: + unique_together = ('user', 'fromMember') + + def __str__(self): + return '{} {}'.format(self.fromMember.collection.uid, self.user) + + class UserInfo(models.Model): owner = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, primary_key=True) version = models.PositiveSmallIntegerField(default=1) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 06bb7a9..54b8d9c 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -262,6 +262,45 @@ class CollectionMemberSerializer(serializers.ModelSerializer): return instance +class CollectionInvitationSerializer(serializers.ModelSerializer): + username = serializers.SlugRelatedField( + source='user', + slug_field=User.USERNAME_FIELD, + queryset=User.objects + ) + collection = serializers.SlugRelatedField( + source='fromMember__collection', + slug_field='uid', + read_only=True, + ) + fromPubkey = BinaryBase64Field(source='fromMember__user__userinfo__pubkey', read_only=True) + signedEncryptionKey = BinaryBase64Field() + + class Meta: + model = models.CollectionInvitation + fields = ('username', 'uid', 'collection', 'signedEncryptionKey', 'accessLevel', 'fromPubkey', 'version') + + def create(self, validated_data): + collection = self.context['collection'] + request = self.context['request'] + + if request.user == validated_data.get('user'): + raise serializers.ValidationError('Inviting yourself is not allowed') + + member = collection.members.get(user=request.user) + + with transaction.atomic(): + return type(self).Meta.model.objects.create(**validated_data, fromMember=member) + + def update(self, instance, validated_data): + with transaction.atomic(): + instance.accessLevel = validated_data.pop('accessLevel') + instance.signedEncryptionKey = validated_data.pop('signedEncryptionKey') + instance.save() + + return instance + + class UserSerializer(serializers.ModelSerializer): class Meta: model = User diff --git a/django_etesync/views.py b/django_etesync/views.py index 5703410..ffb9503 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -34,7 +34,7 @@ import nacl.secret import nacl.hash from . import app_settings, permissions -from .models import Collection, CollectionItem, CollectionItemRevision, CollectionMember +from .models import Collection, CollectionItem, CollectionItemRevision, CollectionMember, CollectionInvitation from .serializers import ( b64encode, AuthenticationSignupSerializer, @@ -48,6 +48,7 @@ from .serializers import ( CollectionItemRevisionSerializer, CollectionItemChunkSerializer, CollectionMemberSerializer, + CollectionInvitationSerializer, UserSerializer, ) @@ -423,6 +424,38 @@ class CollectionMemberViewSet(BaseViewSet): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) +class CollectionInvitationViewSet(BaseViewSet): + allowed_methods = ['GET', 'POST', 'PUT', 'DELETE'] + permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdmin, ) + queryset = CollectionInvitation.objects.all() + serializer_class = CollectionInvitationSerializer + lookup_field = 'uid' + lookup_url_kwarg = 'invitation_uid' + + def get_serializer_context(self): + context = super().get_serializer_context() + collection_uid = self.kwargs['collection_uid'] + try: + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + except Collection.DoesNotExist: + raise Http404('Collection does not exist') + + context.update({'request': self.request, 'collection': collection}) + return context + + def get_queryset(self, queryset=None): + collection_uid = self.kwargs['collection_uid'] + try: + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + except Collection.DoesNotExist: + raise Http404('Collection does not exist') + + if queryset is None: + queryset = type(self).queryset + + return queryset.filter(fromMember__collection=collection) + + class AuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] @@ -561,6 +594,7 @@ class TestAuthenticationViewSet(viewsets.ViewSet): # Delete all of the journal data for this user for a clear test env request.user.collection_set.all().delete() + request.user.incoming_invitations.all().delete() # FIXME: also delete chunk files!!! From 47e1eec122eec8c3cdba484ba098f3327e9551fe Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 20 May 2020 15:15:24 +0300 Subject: [PATCH 147/511] Incoming invitations: implement incoming invitations and accepting them --- django_etesync/models.py | 5 +++++ django_etesync/serializers.py | 38 +++++++++++++++++++++++++++++------ django_etesync/views.py | 26 ++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 6 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index e0a79f6..cbfa269 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -150,6 +150,7 @@ class CollectionInvitation(models.Model): version = models.PositiveSmallIntegerField(default=1) fromMember = models.ForeignKey(CollectionMember, on_delete=models.CASCADE) # FIXME: make sure to delete all invitations for the same collection once one is accepted + # Make sure to not allow invitations if already a member user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name='incoming_invitations', on_delete=models.CASCADE) signedEncryptionKey = models.BinaryField(editable=False, blank=False, null=False) @@ -165,6 +166,10 @@ class CollectionInvitation(models.Model): def __str__(self): return '{} {}'.format(self.fromMember.collection.uid, self.user) + @cached_property + def collection(self): + return self.fromMember.collection + class UserInfo(models.Model): owner = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, primary_key=True) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 54b8d9c..185eb6e 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -268,18 +268,20 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): slug_field=User.USERNAME_FIELD, queryset=User.objects ) - collection = serializers.SlugRelatedField( - source='fromMember__collection', - slug_field='uid', - read_only=True, - ) - fromPubkey = BinaryBase64Field(source='fromMember__user__userinfo__pubkey', read_only=True) + collection = serializers.SerializerMethodField('get_collection') + fromPubkey = serializers.SerializerMethodField('get_from_pubkey') signedEncryptionKey = BinaryBase64Field() class Meta: model = models.CollectionInvitation fields = ('username', 'uid', 'collection', 'signedEncryptionKey', 'accessLevel', 'fromPubkey', 'version') + def get_collection(self, obj): + return obj.collection.uid + + def get_from_pubkey(self, obj): + return b64encode(obj.fromMember.user.userinfo.pubkey) + def create(self, validated_data): collection = self.context['collection'] request = self.context['request'] @@ -301,6 +303,30 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): return instance +class InvitationAcceptSerializer(serializers.Serializer): + encryptionKey = BinaryBase64Field() + + def create(self, validated_data): + + with transaction.atomic(): + invitation = self.context['invitation'] + encryption_key = validated_data.get('encryptionKey') + + member = models.CollectionMember.objects.create( + collection=invitation.collection, + user=invitation.user, + accessLevel=invitation.accessLevel, + encryptionKey=encryption_key, + ) + + invitation.delete() + + return member + + def update(self, instance, validated_data): + raise NotImplementedError() + + class UserSerializer(serializers.ModelSerializer): class Meta: model = User diff --git a/django_etesync/views.py b/django_etesync/views.py index ffb9503..a1d6d09 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -49,6 +49,7 @@ from .serializers import ( CollectionItemChunkSerializer, CollectionMemberSerializer, CollectionInvitationSerializer, + InvitationAcceptSerializer, UserSerializer, ) @@ -456,6 +457,31 @@ class CollectionInvitationViewSet(BaseViewSet): return queryset.filter(fromMember__collection=collection) +class InvitationIncomingViewSet(BaseViewSet): + allowed_methods = ['GET', 'DELETE'] + queryset = CollectionInvitation.objects.all() + serializer_class = CollectionInvitationSerializer + lookup_field = 'uid' + lookup_url_kwarg = 'invitation_uid' + + def get_queryset(self, queryset=None): + if queryset is None: + queryset = type(self).queryset + + return queryset.filter(user=self.request.user) + + @action_decorator(detail=True, allowed_methods=['POST'], methods=['POST']) + def accept(self, request, invitation_uid=None): + invitation = get_object_or_404(self.get_queryset(), uid=invitation_uid) + context = self.get_serializer_context() + context.update({'invitation': invitation}) + + serializer = InvitationAcceptSerializer(data=request.data, context=context) + serializer.is_valid(raise_exception=True) + serializer.save() + return Response(status=status.HTTP_201_CREATED) + + class AuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] From 40b7edcb84794ab9eb4a70eb4cf55a9b6985816e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 24 May 2020 15:20:55 +0300 Subject: [PATCH 148/511] Add a way to fetch a user's pubkey. --- django_etesync/serializers.py | 8 ++++++++ django_etesync/views.py | 11 ++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 185eb6e..1cc4a7e 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -339,6 +339,14 @@ class UserQuerySerializer(serializers.ModelSerializer): fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) +class UserInfoPubkeySerializer(serializers.ModelSerializer): + pubkey = BinaryBase64Field() + + class Meta: + model = models.UserInfo + fields = ('pubkey', ) + + class AuthenticationSignupSerializer(serializers.Serializer): user = UserQuerySerializer(many=False) salt = BinaryBase64Field() diff --git a/django_etesync/views.py b/django_etesync/views.py index a1d6d09..7130eae 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -34,7 +34,7 @@ import nacl.secret import nacl.hash from . import app_settings, permissions -from .models import Collection, CollectionItem, CollectionItemRevision, CollectionMember, CollectionInvitation +from .models import Collection, CollectionItem, CollectionItemRevision, CollectionMember, CollectionInvitation, UserInfo from .serializers import ( b64encode, AuthenticationSignupSerializer, @@ -50,6 +50,7 @@ from .serializers import ( CollectionMemberSerializer, CollectionInvitationSerializer, InvitationAcceptSerializer, + UserInfoPubkeySerializer, UserSerializer, ) @@ -456,6 +457,14 @@ class CollectionInvitationViewSet(BaseViewSet): return queryset.filter(fromMember__collection=collection) + @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) + def fetch_user_profile(self, request, collection_uid=None): + username = request.GET.get('username') + kwargs = {'owner__' + User.USERNAME_FIELD: username} + user_info = get_object_or_404(UserInfo.objects.all(), **kwargs) + serializer = UserInfoPubkeySerializer(user_info) + return Response(serializer.data) + class InvitationIncomingViewSet(BaseViewSet): allowed_methods = ['GET', 'DELETE'] From 7f7d223b9b10cdc7dc2418befe060ae5deb83f30 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 24 May 2020 17:22:43 +0300 Subject: [PATCH 149/511] Fix indentation error. --- django_etesync/serializers.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 1cc4a7e..21ce6a4 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -313,11 +313,11 @@ class InvitationAcceptSerializer(serializers.Serializer): encryption_key = validated_data.get('encryptionKey') member = models.CollectionMember.objects.create( - collection=invitation.collection, - user=invitation.user, - accessLevel=invitation.accessLevel, - encryptionKey=encryption_key, - ) + collection=invitation.collection, + user=invitation.user, + accessLevel=invitation.accessLevel, + encryptionKey=encryption_key, + ) invitation.delete() From 118dbea4e34bbdfc06fec9b6b42d219a10369ce0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 24 May 2020 17:52:09 +0300 Subject: [PATCH 150/511] InvitationSerializer: fix user validator. --- django_etesync/serializers.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 21ce6a4..95dbc1f 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -282,13 +282,16 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): def get_from_pubkey(self, obj): return b64encode(obj.fromMember.user.userinfo.pubkey) - def create(self, validated_data): - collection = self.context['collection'] + def validate_user(self, value): request = self.context['request'] - if request.user == validated_data.get('user'): + if request.user == value: raise serializers.ValidationError('Inviting yourself is not allowed') + def create(self, validated_data): + collection = self.context['collection'] + request = self.context['request'] + member = collection.members.get(user=request.user) with transaction.atomic(): From a965a76c36081958ee9f3a495ac295e0240f9045 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 24 May 2020 18:19:22 +0300 Subject: [PATCH 151/511] Invitation: move outgoing invitations to invite/outgoing. --- django_etesync/permissions.py | 8 +++++-- django_etesync/serializers.py | 15 +++++-------- django_etesync/views.py | 42 +++++++++++++++++++++-------------- 3 files changed, 36 insertions(+), 29 deletions(-) diff --git a/django_etesync/permissions.py b/django_etesync/permissions.py index 29806c6..c371743 100644 --- a/django_etesync/permissions.py +++ b/django_etesync/permissions.py @@ -16,6 +16,11 @@ from rest_framework import permissions from django_etesync.models import Collection, AccessLevels +def is_collection_admin(collection, user): + member = collection.members.filter(user=user).first() + return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) + + class IsCollectionAdmin(permissions.BasePermission): """ Custom permission to only allow owners of a collection to view it @@ -27,8 +32,7 @@ class IsCollectionAdmin(permissions.BasePermission): collection_uid = view.kwargs['collection_uid'] try: collection = view.get_collection_queryset().get(uid=collection_uid) - member = collection.members.filter(user=request.user).first() - return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) + return is_collection_admin(collection, request.user) except Collection.DoesNotExist: # If the collection does not exist, we want to 404 later, not permission denied. return True diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 95dbc1f..29068fa 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -268,29 +268,24 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): slug_field=User.USERNAME_FIELD, queryset=User.objects ) - collection = serializers.SerializerMethodField('get_collection') - fromPubkey = serializers.SerializerMethodField('get_from_pubkey') + collection = serializers.CharField(source='collection.uid') + fromPubkey = BinaryBase64Field(source='fromMember.user.userinfo.pubkey', read_only=True) signedEncryptionKey = BinaryBase64Field() class Meta: model = models.CollectionInvitation fields = ('username', 'uid', 'collection', 'signedEncryptionKey', 'accessLevel', 'fromPubkey', 'version') - def get_collection(self, obj): - return obj.collection.uid - - def get_from_pubkey(self, obj): - return b64encode(obj.fromMember.user.userinfo.pubkey) - def validate_user(self, value): request = self.context['request'] - if request.user == value: + if request.user == value.lower(): raise serializers.ValidationError('Inviting yourself is not allowed') + return value def create(self, validated_data): - collection = self.context['collection'] request = self.context['request'] + collection = validated_data.pop('collection') member = collection.members.get(user=request.user) diff --git a/django_etesync/views.py b/django_etesync/views.py index 7130eae..bde22e4 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -16,6 +16,7 @@ import json from django.conf import settings from django.contrib.auth import get_user_model +from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError from django.db.models import Max from django.http import HttpResponseBadRequest, HttpResponse, Http404 @@ -426,9 +427,9 @@ class CollectionMemberViewSet(BaseViewSet): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) -class CollectionInvitationViewSet(BaseViewSet): +class InvitationOutgoingViewSet(BaseViewSet): allowed_methods = ['GET', 'POST', 'PUT', 'DELETE'] - permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdmin, ) + permission_classes = BaseViewSet.permission_classes queryset = CollectionInvitation.objects.all() serializer_class = CollectionInvitationSerializer lookup_field = 'uid' @@ -436,29 +437,36 @@ class CollectionInvitationViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - collection_uid = self.kwargs['collection_uid'] - try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) - except Collection.DoesNotExist: - raise Http404('Collection does not exist') - - context.update({'request': self.request, 'collection': collection}) + context.update({'request': self.request}) return context def get_queryset(self, queryset=None): - collection_uid = self.kwargs['collection_uid'] - try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) - except Collection.DoesNotExist: - raise Http404('Collection does not exist') - if queryset is None: queryset = type(self).queryset - return queryset.filter(fromMember__collection=collection) + return queryset.filter(fromMember__user=self.request.user) + + def create(self, request, *args, **kwargs): + serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) + if serializer.is_valid(): + collection_uid = serializer.validated_data.get('collection', {}).get('uid') + + try: + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + except Collection.DoesNotExist: + raise Http404('Collection does not exist') + + if not permissions.is_collection_admin(collection, request.user): + raise PermissionDenied('User is not an admin of this collection') + + serializer.save(collection=collection) + + return Response({}, status=status.HTTP_201_CREATED) + + return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) - def fetch_user_profile(self, request, collection_uid=None): + def fetch_user_profile(self, request): username = request.GET.get('username') kwargs = {'owner__' + User.USERNAME_FIELD: username} user_info = get_object_or_404(UserInfo.objects.all(), **kwargs) From 8323f23561fcd58ec2f601f230f4cc4c61202533 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 25 May 2020 17:28:47 +0300 Subject: [PATCH 152/511] Add a nop for api/logout/ It's there for etesync.com and is used to invalidate the token. Unfortunately we can't fully implement it here because the token implementation is lacking. This will be fixed soon once we update the token library with the next version of the protocol. --- etesync_server/urls.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/etesync_server/urls.py b/etesync_server/urls.py index 6a079de..feb8892 100644 --- a/etesync_server/urls.py +++ b/etesync_server/urls.py @@ -34,6 +34,8 @@ from django.views.generic import TemplateView from rest_framework_nested import routers from rest_framework.authtoken import views as token_views +from rest_framework.decorators import api_view +from rest_framework.response import Response from journal import views @@ -47,11 +49,17 @@ journals_router.register(r'members', views.MembersViewSet, basename='journal-mem journals_router.register(r'entries', views.EntryViewSet, basename='journal-entries') +@api_view(['POST']) +def nop_view(request): + return Response({}) + + urlpatterns = [ re_path(r'^api/v1/', include(router.urls)), re_path(r'^api/v1/', include(journals_router.urls)), re_path(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), re_path(r'^api-token-auth/', token_views.obtain_auth_token), + path('api/logout/', nop_view), path('admin/', admin.site.urls), path('', TemplateView.as_view(template_name='success.html')), ] From 2412c295de6a60393573ed65160fcf8a1f793096 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 13:17:35 +0300 Subject: [PATCH 153/511] Signup: fix bug making signup not to work. --- django_etesync/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 29068fa..650b981 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -357,7 +357,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): pubkey = validated_data.pop('pubkey') with transaction.atomic(): - instance = User.objects.get_or_create(**user_data) + instance, _ = User.objects.get_or_create(**user_data) if hasattr(instance, 'userinfo'): raise serializers.ValidationError('User already exists') From 863c405802ecb5c8cdc394c94b76b2a673864cb1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 13:23:45 +0300 Subject: [PATCH 154/511] Rename pubkey to loginPubkey because we'll soon have another pubkey. This breaks sharing because we no longer have a normal pubkey. This will be fixed in the next commit. --- .../migrations/0005_auto_20200526_1021.py | 18 ++++++++++++++++++ django_etesync/models.py | 2 +- django_etesync/serializers.py | 6 ++---- django_etesync/views.py | 2 +- 4 files changed, 22 insertions(+), 6 deletions(-) create mode 100644 django_etesync/migrations/0005_auto_20200526_1021.py diff --git a/django_etesync/migrations/0005_auto_20200526_1021.py b/django_etesync/migrations/0005_auto_20200526_1021.py new file mode 100644 index 0000000..470556b --- /dev/null +++ b/django_etesync/migrations/0005_auto_20200526_1021.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-05-26 10:21 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0004_collectioninvitation_version'), + ] + + operations = [ + migrations.RenameField( + model_name='userinfo', + old_name='pubkey', + new_name='loginPubkey', + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index cbfa269..62c3868 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -174,7 +174,7 @@ class CollectionInvitation(models.Model): class UserInfo(models.Model): owner = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, primary_key=True) version = models.PositiveSmallIntegerField(default=1) - pubkey = models.BinaryField(editable=True, blank=False, null=False) + loginPubkey = models.BinaryField(editable=True, blank=False, null=False) salt = models.BinaryField(editable=True, blank=False, null=False) def __str__(self): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 650b981..5772000 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -348,13 +348,11 @@ class UserInfoPubkeySerializer(serializers.ModelSerializer): class AuthenticationSignupSerializer(serializers.Serializer): user = UserQuerySerializer(many=False) salt = BinaryBase64Field() - pubkey = BinaryBase64Field() + loginPubkey = BinaryBase64Field() def create(self, validated_data): """Function that's called when this serializer creates an item""" user_data = validated_data.pop('user') - salt = validated_data.pop('salt') - pubkey = validated_data.pop('pubkey') with transaction.atomic(): instance, _ = User.objects.get_or_create(**user_data) @@ -364,7 +362,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): instance.set_unusable_password() # FIXME: send email verification - models.UserInfo.objects.create(salt=salt, pubkey=pubkey, owner=instance) + models.UserInfo.objects.create(**validated_data, owner=instance) return instance diff --git a/django_etesync/views.py b/django_etesync/views.py index bde22e4..916c2a2 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -603,7 +603,7 @@ class AuthenticationViewSet(viewsets.ViewSet): content = {'code': 'wrong_host', 'detail': detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) - verify_key = nacl.signing.VerifyKey(user.userinfo.pubkey, encoder=nacl.encoding.RawEncoder) + verify_key = nacl.signing.VerifyKey(user.userinfo.loginPubkey, encoder=nacl.encoding.RawEncoder) verify_key.verify(response_raw, signature) data = self.login_response_data(user) From e94e2f9d70224301f34b388578c68b951e74a72e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 13:44:40 +0300 Subject: [PATCH 155/511] Add a separate pubkey/privatekey for sharing. It's separated from the login one so that encryption key and identity can be rotated separately. --- .../migrations/0006_auto_20200526_1040.py | 25 +++++++++++++++++++ django_etesync/models.py | 2 ++ django_etesync/serializers.py | 7 +++++- 3 files changed, 33 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0006_auto_20200526_1040.py diff --git a/django_etesync/migrations/0006_auto_20200526_1040.py b/django_etesync/migrations/0006_auto_20200526_1040.py new file mode 100644 index 0000000..84e8fa3 --- /dev/null +++ b/django_etesync/migrations/0006_auto_20200526_1040.py @@ -0,0 +1,25 @@ +# Generated by Django 3.0.3 on 2020-05-26 10:40 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0005_auto_20200526_1021'), + ] + + operations = [ + migrations.AddField( + model_name='userinfo', + name='encryptedSeckey', + field=models.BinaryField(default=b'', editable=True), + preserve_default=False, + ), + migrations.AddField( + model_name='userinfo', + name='pubkey', + field=models.BinaryField(default=b'', editable=True), + preserve_default=False, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 62c3868..edfb18d 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -175,6 +175,8 @@ class UserInfo(models.Model): owner = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, primary_key=True) version = models.PositiveSmallIntegerField(default=1) loginPubkey = models.BinaryField(editable=True, blank=False, null=False) + pubkey = models.BinaryField(editable=True, blank=False, null=False) + encryptedSeckey = models.BinaryField(editable=True, blank=False, null=False) salt = models.BinaryField(editable=True, blank=False, null=False) def __str__(self): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 5772000..bff20ad 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -326,9 +326,12 @@ class InvitationAcceptSerializer(serializers.Serializer): class UserSerializer(serializers.ModelSerializer): + pubkey = BinaryBase64Field(source='userinfo.pubkey') + encryptedSeckey = BinaryBase64Field(source='userinfo.encryptedSeckey') + class Meta: model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) + fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, 'pubkey', 'encryptedSeckey') class UserQuerySerializer(serializers.ModelSerializer): @@ -349,6 +352,8 @@ class AuthenticationSignupSerializer(serializers.Serializer): user = UserQuerySerializer(many=False) salt = BinaryBase64Field() loginPubkey = BinaryBase64Field() + pubkey = BinaryBase64Field() + encryptedSeckey = BinaryBase64Field() def create(self, validated_data): """Function that's called when this serializer creates an item""" From 10b9d33ffe95c4ab68ee8a784c1517a3327a6fcf Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 16:13:18 +0300 Subject: [PATCH 156/511] UidValidator: fix to actually validate. --- django_etesync/models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index edfb18d..98f57f2 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -21,7 +21,7 @@ from django.utils.functional import cached_property Base64Url256BitValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{43}$', message='Expected a 256bit base64url.') -UidValidator = RegexValidator(regex=r'[a-zA-Z0-9]', message='Not a valid UID') +UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9]*$', message='Not a valid UID') class Collection(models.Model): From fce844bfc3c7078167b2e34edf1b0b9696197807 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 16:26:57 +0300 Subject: [PATCH 157/511] Uid: Change how validation is done. --- .../migrations/0007_auto_20200526_1336.py | 39 +++++++++++++++++++ django_etesync/models.py | 12 +++--- 2 files changed, 45 insertions(+), 6 deletions(-) create mode 100644 django_etesync/migrations/0007_auto_20200526_1336.py diff --git a/django_etesync/migrations/0007_auto_20200526_1336.py b/django_etesync/migrations/0007_auto_20200526_1336.py new file mode 100644 index 0000000..37e31ac --- /dev/null +++ b/django_etesync/migrations/0007_auto_20200526_1336.py @@ -0,0 +1,39 @@ +# Generated by Django 3.0.3 on 2020-05-26 13:36 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0006_auto_20200526_1040'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9]*$')]), + ), + migrations.AlterField( + model_name='collectioninvitation', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')]), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=43, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9]*$')]), + ), + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')]), + ), + migrations.AlterField( + model_name='collectionitemrevision', + name='uid', + field=models.CharField(db_index=True, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')]), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 98f57f2..3a286ce 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -20,13 +20,13 @@ from django.core.validators import RegexValidator from django.utils.functional import cached_property -Base64Url256BitValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{43}$', message='Expected a 256bit base64url.') +Base64Url256BitlValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{42,43}$', message='Expected a base64url.') UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9]*$', message='Not a valid UID') class Collection(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, - max_length=44, validators=[UidValidator]) + max_length=43, validators=[UidValidator]) version = models.PositiveSmallIntegerField() owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) @@ -60,7 +60,7 @@ class Collection(models.Model): class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, null=True, - max_length=44, validators=[UidValidator]) + max_length=43, validators=[UidValidator]) collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) version = models.PositiveSmallIntegerField() encryptionKey = models.BinaryField(editable=True, blank=False, null=True) @@ -90,7 +90,7 @@ def chunk_directory_path(instance, filename): class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, - max_length=44, validators=[Base64Url256BitValidator]) + max_length=43, validators=[Base64Url256BitlValidator]) item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) @@ -100,7 +100,7 @@ class CollectionItemChunk(models.Model): class CollectionItemRevision(models.Model): uid = models.CharField(db_index=True, unique=True, blank=False, null=False, - max_length=44, validators=[Base64Url256BitValidator]) + max_length=43, validators=[Base64Url256BitlValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) @@ -146,7 +146,7 @@ class CollectionMember(models.Model): class CollectionInvitation(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, - max_length=44, validators=[Base64Url256BitValidator]) + max_length=43, validators=[Base64Url256BitlValidator]) version = models.PositiveSmallIntegerField(default=1) fromMember = models.ForeignKey(CollectionMember, on_delete=models.CASCADE) # FIXME: make sure to delete all invitations for the same collection once one is accepted From 3cdb7783fe6ac2cfcc9840c6e4e6d9c39dd4f917 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 18:14:39 +0300 Subject: [PATCH 158/511] Make sure to always return fresh stokens. --- django_etesync/models.py | 4 ++-- django_etesync/views.py | 34 ++++++++++++++++++++++++++++++---- 2 files changed, 32 insertions(+), 6 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 3a286ce..ba6665c 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -44,7 +44,7 @@ class Collection(models.Model): def content(self): return self.main_item.content - @cached_property + @property def stoken(self): return self.main_item.stoken @@ -75,7 +75,7 @@ class CollectionItem(models.Model): def content(self): return self.revisions.get(current=True) - @cached_property + @property def stoken(self): return self.content.uid diff --git a/django_etesync/views.py b/django_etesync/views.py index 916c2a2..7f07f2e 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -59,6 +59,24 @@ from .serializers import ( User = get_user_model() +def get_fresh_stoken(obj): + try: + del obj.main_item + except AttributeError: + pass + + return obj.stoken + + +def get_fresh_item_stoken(obj): + try: + del obj.content + except AttributeError: + pass + + return obj.stoken + + class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) @@ -141,16 +159,24 @@ class CollectionViewSet(BaseViewSet): def partial_update(self, request, uid=None): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + def update(self, request, *args, **kwargs): + instance = self.get_object() + serializer = self.get_serializer(instance, data=request.data) + serializer.is_valid(raise_exception=True) + self.perform_update(serializer) + + return Response({'stoken': get_fresh_stoken(instance)}) + def create(self, request, *args, **kwargs): serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) if serializer.is_valid(): try: - serializer.save(owner=self.request.user) + instance = serializer.save(owner=self.request.user) except IntegrityError: content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) - return Response({}, status=status.HTTP_201_CREATED) + return Response({'stoken': get_fresh_stoken(instance)}, status=status.HTTP_201_CREATED) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) @@ -304,7 +330,7 @@ class CollectionItemViewSet(BaseViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) ret = { - "data": [item.stoken for item in items], + "data": [get_fresh_item_stoken(item) for item in items], } return Response(ret, status=status.HTTP_200_OK) @@ -343,7 +369,7 @@ class CollectionItemViewSet(BaseViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) ret = { - "data": [item.stoken for item in items], + "data": [get_fresh_item_stoken(item) for item in items], } return Response(ret, status=status.HTTP_200_OK) From 2a39f3538e2d846d2f3051100ea672b248f27b64 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 18:52:44 +0300 Subject: [PATCH 159/511] Change to standalone stoken objects (+ small optimisation). Makes it possible to now generate Stokens as we need so we can add them to non-revision objects, for example, membership changes. We also slightly improved how we filter by revs. --- django_etesync/models.py | 15 +++++++++++++-- django_etesync/serializers.py | 4 +++- django_etesync/views.py | 26 +++++++++++++++++--------- 3 files changed, 33 insertions(+), 12 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index ba6665c..3767b3a 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -18,6 +18,7 @@ from django.db import models from django.conf import settings from django.core.validators import RegexValidator from django.utils.functional import cached_property +from django.utils.crypto import get_random_string Base64Url256BitlValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{42,43}$', message='Expected a base64url.') @@ -55,7 +56,7 @@ class Collection(models.Model): # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. return None - return last_revision.uid + return last_revision.stoken.uid class CollectionItem(models.Model): @@ -77,7 +78,7 @@ class CollectionItem(models.Model): @property def stoken(self): - return self.content.uid + return self.content.stoken.uid def chunk_directory_path(instance, filename): @@ -98,7 +99,17 @@ class CollectionItemChunk(models.Model): return self.uid +def generate_stoken_uid(): + return get_random_string(32, allowed_chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_') + + +class Stoken(models.Model): + uid = models.CharField(db_index=True, unique=True, blank=False, null=False, default=generate_stoken_uid, + max_length=43, validators=[Base64Url256BitlValidator]) + + class CollectionItemRevision(models.Model): + stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) uid = models.CharField(db_index=True, unique=True, blank=False, null=False, max_length=43, validators=[Base64Url256BitlValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index bff20ad..d08fcfd 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -38,7 +38,9 @@ def process_revisions_for_item(item, revision_data): chunk = models.CollectionItemChunk.objects.get(uid=uid) chunks_objs.append(chunk) - revision = models.CollectionItemRevision.objects.create(**revision_data, item=item) + stoken = models.Stoken.objects.create() + + revision = models.CollectionItemRevision.objects.create(**revision_data, item=item, stoken=stoken) for chunk in chunks_objs: models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) return revision diff --git a/django_etesync/views.py b/django_etesync/views.py index 7f07f2e..49b8110 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -35,7 +35,15 @@ import nacl.secret import nacl.hash from . import app_settings, permissions -from .models import Collection, CollectionItem, CollectionItemRevision, CollectionMember, CollectionInvitation, UserInfo +from .models import ( + Collection, + CollectionItem, + CollectionItemRevision, + CollectionMember, + CollectionInvitation, + Stoken, + UserInfo, + ) from .serializers import ( b64encode, AuthenticationSignupSerializer, @@ -94,18 +102,18 @@ class BaseViewSet(viewsets.ModelViewSet): user = self.request.user return queryset.filter(members__user=user) - def get_cstoken_rev(self, request): + def get_cstoken_obj(self, request): cstoken = request.GET.get('cstoken', None) if cstoken is not None: - return get_object_or_404(CollectionItemRevision.objects.all(), uid=cstoken) + return get_object_or_404(Stoken.objects.all(), uid=cstoken) return None def filter_by_cstoken(self, request, queryset): cstoken_id_field = self.cstoken_id_field + '__id' - cstoken_rev = self.get_cstoken_rev(request) + cstoken_rev = self.get_cstoken_obj(request) if cstoken_rev is not None: filter_by = {cstoken_id_field + '__gt': cstoken_rev.id} queryset = queryset.filter(**filter_by) @@ -116,7 +124,7 @@ class BaseViewSet(viewsets.ModelViewSet): cstoken_id_field = self.cstoken_id_field + '__id' new_cstoken_id = queryset.aggregate(cstoken_id=Max(cstoken_id_field))['cstoken_id'] - new_cstoken = new_cstoken_id and CollectionItemRevision.objects.get(id=new_cstoken_id).uid + new_cstoken = new_cstoken_id and Stoken.objects.get(id=new_cstoken_id).uid return queryset, new_cstoken @@ -139,7 +147,7 @@ class CollectionViewSet(BaseViewSet): queryset = Collection.objects.all() serializer_class = CollectionSerializer lookup_field = 'uid' - cstoken_id_field = 'items__revisions' + cstoken_id_field = 'items__revisions__stoken' def get_queryset(self, queryset=None): if queryset is None: @@ -199,7 +207,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = 'uid' - cstoken_id_field = 'revisions' + cstoken_id_field = 'revisions__stoken' def get_queryset(self): collection_uid = self.kwargs['collection_uid'] @@ -290,8 +298,8 @@ class CollectionItemViewSet(BaseViewSet): queryset, cstoken_rev = self.filter_by_cstoken(request, queryset) uids, stokens = zip(*[(item['uid'], item.get('stoken')) for item in serializer.validated_data]) - rev_ids = CollectionItemRevision.objects.filter(uid__in=stokens, current=True).values_list('id', flat=True) - queryset = queryset.filter(uid__in=uids).exclude(revisions__id__in=rev_ids) + revs = CollectionItemRevision.objects.filter(stoken__uid__in=stokens, current=True) + queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) queryset, new_cstoken = self.get_queryset_cstoken(queryset) cstoken = cstoken_rev and cstoken_rev.uid From 6e7ad92a12365d7c2c32cfd2f7add657464c9b6f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 26 May 2020 18:52:44 +0300 Subject: [PATCH 160/511] Add missing migrations forgotten in the previous commit Missing from: 73f4ff765c7713c9aa48dec2bfc4c3c1c0c7e9f3 --- .../migrations/0008_auto_20200526_1535.py | 28 +++++++++++++++++++ .../migrations/0009_auto_20200526_1535.py | 23 +++++++++++++++ .../migrations/0010_auto_20200526_1539.py | 19 +++++++++++++ 3 files changed, 70 insertions(+) create mode 100644 django_etesync/migrations/0008_auto_20200526_1535.py create mode 100644 django_etesync/migrations/0009_auto_20200526_1535.py create mode 100644 django_etesync/migrations/0010_auto_20200526_1539.py diff --git a/django_etesync/migrations/0008_auto_20200526_1535.py b/django_etesync/migrations/0008_auto_20200526_1535.py new file mode 100644 index 0000000..e544bdd --- /dev/null +++ b/django_etesync/migrations/0008_auto_20200526_1535.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.3 on 2020-05-26 15:35 + +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion +import django_etesync.models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0007_auto_20200526_1336'), + ] + + operations = [ + migrations.CreateModel( + name='Stoken', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.CharField(db_index=True, default=django_etesync.models.generate_stoken_uid, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')])), + ], + ), + migrations.AddField( + model_name='collectionitemrevision', + name='stoken', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken'), + ), + ] diff --git a/django_etesync/migrations/0009_auto_20200526_1535.py b/django_etesync/migrations/0009_auto_20200526_1535.py new file mode 100644 index 0000000..53100b3 --- /dev/null +++ b/django_etesync/migrations/0009_auto_20200526_1535.py @@ -0,0 +1,23 @@ +# Generated by Django 3.0.3 on 2020-05-26 15:35 + +from django.db import migrations + + +def create_stokens(apps, schema_editor): + Stoken = apps.get_model('django_etesync', 'Stoken') + CollectionItemRevision = apps.get_model('django_etesync', 'CollectionItemRevision') + + for rev in CollectionItemRevision.objects.all(): + rev.stoken = Stoken.objects.create() + rev.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0008_auto_20200526_1535'), + ] + + operations = [ + migrations.RunPython(create_stokens), + ] diff --git a/django_etesync/migrations/0010_auto_20200526_1539.py b/django_etesync/migrations/0010_auto_20200526_1539.py new file mode 100644 index 0000000..c894fd2 --- /dev/null +++ b/django_etesync/migrations/0010_auto_20200526_1539.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-05-26 15:39 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0009_auto_20200526_1535'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemrevision', + name='stoken', + field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken'), + ), + ] From 8eee280bbb41f114b6082e3798803d9c36028451 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 09:39:31 +0300 Subject: [PATCH 161/511] Split cstoken and stoken to be different concepts The stokens are really just integrity checks for items, and are really just tied to what revision we expected to have first what we have. So we will rename stoken to lastRev or something, and have them completely separate. A partial revert of e22a49f982046e875d4e1c5007a91353527d7a0f --- django_etesync/models.py | 6 +++--- django_etesync/views.py | 28 ++++------------------------ 2 files changed, 7 insertions(+), 27 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 3767b3a..6c50ea2 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -41,13 +41,13 @@ class Collection(models.Model): def main_item(self): return self.items.get(uid=None) - @cached_property + @property def content(self): return self.main_item.content @property def stoken(self): - return self.main_item.stoken + return self.content.uid @cached_property def cstoken(self): @@ -78,7 +78,7 @@ class CollectionItem(models.Model): @property def stoken(self): - return self.content.stoken.uid + return self.content.uid def chunk_directory_path(instance, filename): diff --git a/django_etesync/views.py b/django_etesync/views.py index 49b8110..2341484 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -67,24 +67,6 @@ from .serializers import ( User = get_user_model() -def get_fresh_stoken(obj): - try: - del obj.main_item - except AttributeError: - pass - - return obj.stoken - - -def get_fresh_item_stoken(obj): - try: - del obj.content - except AttributeError: - pass - - return obj.stoken - - class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) @@ -173,18 +155,18 @@ class CollectionViewSet(BaseViewSet): serializer.is_valid(raise_exception=True) self.perform_update(serializer) - return Response({'stoken': get_fresh_stoken(instance)}) + return Response({}) def create(self, request, *args, **kwargs): serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) if serializer.is_valid(): try: - instance = serializer.save(owner=self.request.user) + serializer.save(owner=self.request.user) except IntegrityError: content = {'code': 'integrity_error'} return Response(content, status=status.HTTP_400_BAD_REQUEST) - return Response({'stoken': get_fresh_stoken(instance)}, status=status.HTTP_201_CREATED) + return Response({}, status=status.HTTP_201_CREATED) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) @@ -298,7 +280,7 @@ class CollectionItemViewSet(BaseViewSet): queryset, cstoken_rev = self.filter_by_cstoken(request, queryset) uids, stokens = zip(*[(item['uid'], item.get('stoken')) for item in serializer.validated_data]) - revs = CollectionItemRevision.objects.filter(stoken__uid__in=stokens, current=True) + revs = CollectionItemRevision.objects.filter(uid__in=stokens, current=True) queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) queryset, new_cstoken = self.get_queryset_cstoken(queryset) @@ -338,7 +320,6 @@ class CollectionItemViewSet(BaseViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) ret = { - "data": [get_fresh_item_stoken(item) for item in items], } return Response(ret, status=status.HTTP_200_OK) @@ -377,7 +358,6 @@ class CollectionItemViewSet(BaseViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) ret = { - "data": [get_fresh_item_stoken(item) for item in items], } return Response(ret, status=status.HTTP_200_OK) From 9c63f8d6746661542982aff7147df8129cfbc1e7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 10:09:45 +0300 Subject: [PATCH 162/511] Rename stoken to etag and cstoken to stoken. This conforms better with what people know from HTTP and properly differentiates from CSToken which is now renamed to stoken. --- django_etesync/models.py | 6 +-- django_etesync/serializers.py | 44 +++++++++--------- django_etesync/views.py | 84 +++++++++++++++++------------------ 3 files changed, 67 insertions(+), 67 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index 6c50ea2..c9c95a1 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -46,11 +46,11 @@ class Collection(models.Model): return self.main_item.content @property - def stoken(self): + def etag(self): return self.content.uid @cached_property - def cstoken(self): + def stoken(self): last_revision = CollectionItemRevision.objects.filter(item__collection=self).last() if last_revision is None: # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. @@ -77,7 +77,7 @@ class CollectionItem(models.Model): return self.revisions.get(current=True) @property - def stoken(self): + def etag(self): return self.content.uid diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index d08fcfd..75baba0 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -114,17 +114,17 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): class CollectionItemSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() - stoken = serializers.CharField(allow_null=True) + etag = serializers.CharField(allow_null=True) content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.CollectionItem - fields = ('uid', 'version', 'encryptionKey', 'content', 'stoken') + fields = ('uid', 'version', 'encryptionKey', 'content', 'etag') def create(self, validated_data): """Function that's called when this serializer creates an item""" - validate_stoken = self.context.get('validate_stoken', False) - stoken = validated_data.pop('stoken') + validate_etag = self.context.get('validate_etag', False) + etag = validated_data.pop('etag') revision_data = validated_data.pop('content') uid = validated_data.pop('uid') @@ -132,10 +132,10 @@ class CollectionItemSerializer(serializers.ModelSerializer): with transaction.atomic(): instance, created = Model.objects.get_or_create(uid=uid, defaults=validated_data) - cur_stoken = instance.stoken if not created else None + cur_etag = instance.etag if not created else None - if validate_stoken and cur_stoken != stoken: - raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(cur_stoken, stoken)) + if validate_etag and cur_etag != etag: + raise serializers.ValidationError('Wrong etag. Expected {} got {}'.format(cur_etag, etag)) if not created: # We don't have to use select_for_update here because the unique constraint on current guards against @@ -154,39 +154,39 @@ class CollectionItemSerializer(serializers.ModelSerializer): class CollectionItemDepSerializer(serializers.ModelSerializer): - stoken = serializers.CharField() + etag = serializers.CharField() class Meta: model = models.CollectionItem - fields = ('uid', 'stoken') + fields = ('uid', 'etag') def validate(self, data): item = self.__class__.Meta.model.objects.get(uid=data['uid']) - stoken = data['stoken'] - if item.stoken != stoken: - raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(item.stoken, stoken)) + etag = data['etag'] + if item.etag != etag: + raise serializers.ValidationError('Wrong etag. Expected {} got {}'.format(item.etag, etag)) return data class CollectionItemBulkGetSerializer(serializers.ModelSerializer): - stoken = serializers.CharField(required=False) + etag = serializers.CharField(required=False) class Meta: model = models.CollectionItem - fields = ('uid', 'stoken') + fields = ('uid', 'etag') class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') - cstoken = serializers.CharField(read_only=True) - stoken = serializers.CharField(allow_null=True) + stoken = serializers.CharField(read_only=True) + etag = serializers.CharField(allow_null=True) content = CollectionItemRevisionSerializer(many=False) class Meta: model = models.Collection - fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'cstoken', 'stoken') + fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'stoken', 'etag') def get_access_level_from_context(self, obj): request = self.context.get('request', None) @@ -196,13 +196,13 @@ class CollectionSerializer(serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" - stoken = validated_data.pop('stoken') + etag = validated_data.pop('etag') revision_data = validated_data.pop('content') encryption_key = validated_data.pop('encryptionKey') instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): - if stoken is not None: + if etag is not None: raise serializers.ValidationError('Stoken is not None') instance.save() @@ -221,12 +221,12 @@ class CollectionSerializer(serializers.ModelSerializer): def update(self, instance, validated_data): """Function that's called when this serializer is meant to update an item""" - stoken = validated_data.pop('stoken') + etag = validated_data.pop('etag') revision_data = validated_data.pop('content') with transaction.atomic(): - if stoken != instance.stoken: - raise serializers.ValidationError('Wrong stoken. Expected {} got {}'.format(instance.stoken, stoken)) + if etag != instance.etag: + raise serializers.ValidationError('Wrong etag. Expected {} got {}'.format(instance.etag, etag)) main_item = instance.main_item # We don't have to use select_for_update here because the unique constraint on current guards against diff --git a/django_etesync/views.py b/django_etesync/views.py index 2341484..e66bb82 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -70,7 +70,7 @@ User = get_user_model() class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) - cstoken_id_field = None + stoken_id_field = None def get_serializer_class(self): serializer_class = self.serializer_class @@ -84,43 +84,43 @@ class BaseViewSet(viewsets.ModelViewSet): user = self.request.user return queryset.filter(members__user=user) - def get_cstoken_obj(self, request): - cstoken = request.GET.get('cstoken', None) + def get_stoken_obj(self, request): + stoken = request.GET.get('stoken', None) - if cstoken is not None: - return get_object_or_404(Stoken.objects.all(), uid=cstoken) + if stoken is not None: + return get_object_or_404(Stoken.objects.all(), uid=stoken) return None - def filter_by_cstoken(self, request, queryset): - cstoken_id_field = self.cstoken_id_field + '__id' + def filter_by_stoken(self, request, queryset): + stoken_id_field = self.stoken_id_field + '__id' - cstoken_rev = self.get_cstoken_obj(request) - if cstoken_rev is not None: - filter_by = {cstoken_id_field + '__gt': cstoken_rev.id} + stoken_rev = self.get_stoken_obj(request) + if stoken_rev is not None: + filter_by = {stoken_id_field + '__gt': stoken_rev.id} queryset = queryset.filter(**filter_by) - return queryset, cstoken_rev + return queryset, stoken_rev - def get_queryset_cstoken(self, queryset): - cstoken_id_field = self.cstoken_id_field + '__id' + def get_queryset_stoken(self, queryset): + stoken_id_field = self.stoken_id_field + '__id' - new_cstoken_id = queryset.aggregate(cstoken_id=Max(cstoken_id_field))['cstoken_id'] - new_cstoken = new_cstoken_id and Stoken.objects.get(id=new_cstoken_id).uid + new_stoken_id = queryset.aggregate(stoken_id=Max(stoken_id_field))['stoken_id'] + new_stoken = new_stoken_id and Stoken.objects.get(id=new_stoken_id).uid - return queryset, new_cstoken + return queryset, new_stoken - def filter_by_cstoken_and_limit(self, request, queryset): + def filter_by_stoken_and_limit(self, request, queryset): limit = int(request.GET.get('limit', 50)) - queryset, cstoken_rev = self.filter_by_cstoken(request, queryset) - cstoken = cstoken_rev.uid if cstoken_rev is not None else None + queryset, stoken_rev = self.filter_by_stoken(request, queryset) + stoken = stoken_rev.uid if stoken_rev is not None else None queryset = queryset[:limit] - queryset, new_cstoken = self.get_queryset_cstoken(queryset) - new_cstoken = new_cstoken or cstoken + queryset, new_stoken = self.get_queryset_stoken(queryset) + new_stoken = new_stoken or stoken - return queryset, new_cstoken + return queryset, new_stoken class CollectionViewSet(BaseViewSet): @@ -129,7 +129,7 @@ class CollectionViewSet(BaseViewSet): queryset = Collection.objects.all() serializer_class = CollectionSerializer lookup_field = 'uid' - cstoken_id_field = 'items__revisions__stoken' + stoken_id_field = 'items__revisions__stoken' def get_queryset(self, queryset=None): if queryset is None: @@ -172,13 +172,13 @@ class CollectionViewSet(BaseViewSet): def list(self, request): queryset = self.get_queryset() - queryset, new_cstoken = self.filter_by_cstoken_and_limit(request, queryset) + queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) ret = { 'data': serializer.data, - 'cstoken': new_cstoken, + 'stoken': new_stoken, } return Response(ret) @@ -189,7 +189,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = 'uid' - cstoken_id_field = 'revisions__stoken' + stoken_id_field = 'revisions__stoken' def get_queryset(self): collection_uid = self.kwargs['collection_uid'] @@ -240,13 +240,13 @@ class CollectionItemViewSet(BaseViewSet): def list(self, request, collection_uid=None): queryset = self.get_queryset() - queryset, new_cstoken = self.filter_by_cstoken_and_limit(request, queryset) + queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) ret = { 'data': serializer.data, - 'cstoken': new_cstoken, + 'stoken': new_stoken, } return Response(ret) @@ -277,21 +277,21 @@ class CollectionItemViewSet(BaseViewSet): 'detail': 'Request has too many items. Limit: {}'. format(item_limit)} return Response(content, status=status.HTTP_400_BAD_REQUEST) - queryset, cstoken_rev = self.filter_by_cstoken(request, queryset) + queryset, stoken_rev = self.filter_by_stoken(request, queryset) - uids, stokens = zip(*[(item['uid'], item.get('stoken')) for item in serializer.validated_data]) - revs = CollectionItemRevision.objects.filter(uid__in=stokens, current=True) + uids, etags = zip(*[(item['uid'], item.get('etag')) for item in serializer.validated_data]) + revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) - queryset, new_cstoken = self.get_queryset_cstoken(queryset) - cstoken = cstoken_rev and cstoken_rev.uid - new_cstoken = new_cstoken or cstoken + queryset, new_stoken = self.get_queryset_stoken(queryset) + stoken = stoken_rev and stoken_rev.uid + new_stoken = new_stoken or stoken serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) ret = { 'data': serializer.data, - 'cstoken': new_cstoken, + 'stoken': new_stoken, } return Response(ret) @@ -299,11 +299,11 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def batch(self, request, collection_uid=None): - cstoken = request.GET.get('cstoken', None) + stoken = request.GET.get('stoken', None) collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - if cstoken is not None and cstoken != collection_object.cstoken: - content = {'code': 'stale_cstoken', 'detail': 'CSToken is too old'} + if stoken is not None and stoken != collection_object.stoken: + content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} return Response(content, status=status.HTTP_400_BAD_REQUEST) items = request.data.get('items') @@ -331,18 +331,18 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def transaction(self, request, collection_uid=None): - cstoken = request.GET.get('cstoken', None) + stoken = request.GET.get('stoken', None) collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - if cstoken is not None and cstoken != collection_object.cstoken: - content = {'code': 'stale_cstoken', 'detail': 'CSToken is too old'} + if stoken is not None and stoken != collection_object.stoken: + content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} return Response(content, status=status.HTTP_400_BAD_REQUEST) items = request.data.get('items') deps = request.data.get('deps', None) # FIXME: It should just be one serializer context = self.get_serializer_context() - context.update({'validate_stoken': True}) + context.update({'validate_etag': True}) serializer = self.get_serializer_class()(data=items, context=context, many=True) deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) From 91aadb65656cbc0676ccdfea9b1b8f7c7d316140 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 10:16:55 +0300 Subject: [PATCH 163/511] Make etag write-only. --- django_etesync/serializers.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 75baba0..f78803a 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -114,7 +114,7 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): class CollectionItemSerializer(serializers.ModelSerializer): encryptionKey = BinaryBase64Field() - etag = serializers.CharField(allow_null=True) + etag = serializers.CharField(allow_null=True, write_only=True) content = CollectionItemRevisionSerializer(many=False) class Meta: @@ -181,7 +181,7 @@ class CollectionSerializer(serializers.ModelSerializer): encryptionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) - etag = serializers.CharField(allow_null=True) + etag = serializers.CharField(allow_null=True, write_only=True) content = CollectionItemRevisionSerializer(many=False) class Meta: From 1f18f4e50bc228f165afacba163e73b339755c7d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 10:52:27 +0300 Subject: [PATCH 164/511] CollectionMember: add stokens when we create/change the member. --- .../0011_collectionmember_stoken.py | 19 +++++++++++++++ .../migrations/0012_auto_20200527_0743.py | 23 +++++++++++++++++++ django_etesync/models.py | 1 + django_etesync/serializers.py | 9 ++++++-- 4 files changed, 50 insertions(+), 2 deletions(-) create mode 100644 django_etesync/migrations/0011_collectionmember_stoken.py create mode 100644 django_etesync/migrations/0012_auto_20200527_0743.py diff --git a/django_etesync/migrations/0011_collectionmember_stoken.py b/django_etesync/migrations/0011_collectionmember_stoken.py new file mode 100644 index 0000000..6b79bf6 --- /dev/null +++ b/django_etesync/migrations/0011_collectionmember_stoken.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-05-27 07:43 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0010_auto_20200526_1539'), + ] + + operations = [ + migrations.AddField( + model_name='collectionmember', + name='stoken', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken'), + ), + ] diff --git a/django_etesync/migrations/0012_auto_20200527_0743.py b/django_etesync/migrations/0012_auto_20200527_0743.py new file mode 100644 index 0000000..28b8745 --- /dev/null +++ b/django_etesync/migrations/0012_auto_20200527_0743.py @@ -0,0 +1,23 @@ +# Generated by Django 3.0.3 on 2020-05-27 07:43 + +from django.db import migrations + + +def create_stokens(apps, schema_editor): + Stoken = apps.get_model('django_etesync', 'Stoken') + CollectionMember = apps.get_model('django_etesync', 'CollectionMember') + + for member in CollectionMember.objects.all(): + member.stoken = Stoken.objects.create() + member.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0011_collectionmember_stoken'), + ] + + operations = [ + migrations.RunPython(create_stokens), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index c9c95a1..ca68209 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -139,6 +139,7 @@ class AccessLevels(models.TextChoices): class CollectionMember(models.Model): + stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT, null=True) collection = models.ForeignKey(Collection, related_name='members', on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index f78803a..7079742 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -212,6 +212,7 @@ class CollectionSerializer(serializers.ModelSerializer): process_revisions_for_item(main_item, revision_data) models.CollectionMember(collection=instance, + stoken=models.Stoken.objects.create(), user=validated_data.get('owner'), accessLevel=models.AccessLevels.ADMIN, encryptionKey=encryption_key, @@ -258,8 +259,11 @@ class CollectionMemberSerializer(serializers.ModelSerializer): def update(self, instance, validated_data): with transaction.atomic(): # We only allow updating accessLevel - instance.accessLevel = validated_data.pop('accessLevel') - instance.save() + access_level = validated_data.pop('accessLevel') + if instance.accessLevel != access_level: + instance.stoken = models.Stoken.objects.create() + instance.accessLevel = access_level + instance.save() return instance @@ -314,6 +318,7 @@ class InvitationAcceptSerializer(serializers.Serializer): member = models.CollectionMember.objects.create( collection=invitation.collection, + stoken=models.Stoken.objects.create(), user=invitation.user, accessLevel=invitation.accessLevel, encryptionKey=encryption_key, From d93a5d3f06ff9f3f140e96fc080a6a655906e84f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 12:13:54 +0300 Subject: [PATCH 165/511] Collections: use the member stokens for filtering based on stoken While at it, also generalised the stoken handling to be generic and extendible. --- django_etesync/models.py | 13 ++++++++----- django_etesync/views.py | 25 +++++++++++++------------ 2 files changed, 21 insertions(+), 17 deletions(-) diff --git a/django_etesync/models.py b/django_etesync/models.py index ca68209..91746ba 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -17,6 +17,7 @@ from pathlib import Path from django.db import models from django.conf import settings from django.core.validators import RegexValidator +from django.db.models import Q from django.utils.functional import cached_property from django.utils.crypto import get_random_string @@ -51,12 +52,14 @@ class Collection(models.Model): @cached_property def stoken(self): - last_revision = CollectionItemRevision.objects.filter(item__collection=self).last() - if last_revision is None: - # FIXME: what is the etag for None? Though if we use the revision for collection it should be shared anyway. - return None + stoken = Stoken.objects.filter( + Q(collectionitemrevision__item__collection=self) | Q(collectionmember__collection=self) + ).order_by('id').last() - return last_revision.stoken.uid + if stoken is None: + raise Exception('stoken is None. Should never happen') + + return stoken.uid class CollectionItem(models.Model): diff --git a/django_etesync/views.py b/django_etesync/views.py index e66bb82..0f448fa 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -13,12 +13,14 @@ # along with this program. If not, see . import json +from functools import reduce from django.conf import settings from django.contrib.auth import get_user_model from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError -from django.db.models import Max +from django.db.models import Max, Q +from django.db.models.functions import Greatest from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -70,7 +72,7 @@ User = get_user_model() class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) - stoken_id_field = None + stoken_id_fields = None def get_serializer_class(self): serializer_class = self.serializer_class @@ -93,20 +95,19 @@ class BaseViewSet(viewsets.ModelViewSet): return None def filter_by_stoken(self, request, queryset): - stoken_id_field = self.stoken_id_field + '__id' - stoken_rev = self.get_stoken_obj(request) if stoken_rev is not None: - filter_by = {stoken_id_field + '__gt': stoken_rev.id} - queryset = queryset.filter(**filter_by) + filter_by_map = map(lambda x: Q(**{x + '__gt': stoken_rev.id}), self.stoken_id_fields) + filter_by = reduce(lambda x, y: x | y, filter_by_map) + queryset = queryset.filter(filter_by).distinct() return queryset, stoken_rev def get_queryset_stoken(self, queryset): - stoken_id_field = self.stoken_id_field + '__id' - - new_stoken_id = queryset.aggregate(stoken_id=Max(stoken_id_field))['stoken_id'] - new_stoken = new_stoken_id and Stoken.objects.get(id=new_stoken_id).uid + aggr_fields = {x: Max(x) for x in self.stoken_id_fields} + aggr = queryset.aggregate(**aggr_fields) + maxid = max(map(lambda x: x or -1, aggr.values())) + new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid).uid return queryset, new_stoken @@ -129,7 +130,7 @@ class CollectionViewSet(BaseViewSet): queryset = Collection.objects.all() serializer_class = CollectionSerializer lookup_field = 'uid' - stoken_id_field = 'items__revisions__stoken' + stoken_id_fields = ['items__revisions__stoken__id', 'members__stoken__id'] def get_queryset(self, queryset=None): if queryset is None: @@ -189,7 +190,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = 'uid' - stoken_id_field = 'revisions__stoken' + stoken_id_fields = ['revisions__stoken__id'] def get_queryset(self): collection_uid = self.kwargs['collection_uid'] From 6e7fd5d0dd03b783cfcf99a1960a79d440988cf7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 16:03:16 +0300 Subject: [PATCH 166/511] Collection membership: implement leaving/revoking access. --- .../0013_collectionmemberremoved.py | 28 +++++++++++++++++++ django_etesync/models.py | 25 ++++++++++++++++- django_etesync/serializers.py | 6 ++-- django_etesync/views.py | 28 ++++++++++++++++++- 4 files changed, 83 insertions(+), 4 deletions(-) create mode 100644 django_etesync/migrations/0013_collectionmemberremoved.py diff --git a/django_etesync/migrations/0013_collectionmemberremoved.py b/django_etesync/migrations/0013_collectionmemberremoved.py new file mode 100644 index 0000000..e796232 --- /dev/null +++ b/django_etesync/migrations/0013_collectionmemberremoved.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.3 on 2020-05-27 11:29 + +from django.conf import settings +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ('django_etesync', '0012_auto_20200527_0743'), + ] + + operations = [ + migrations.CreateModel( + name='CollectionMemberRemoved', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='removed_members', to='django_etesync.Collection')), + ('stoken', models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken')), + ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ], + options={ + 'unique_together': {('user', 'collection')}, + }, + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 91746ba..7fbea61 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -14,7 +14,7 @@ from pathlib import Path -from django.db import models +from django.db import models, transaction from django.conf import settings from django.core.validators import RegexValidator from django.db.models import Q @@ -158,6 +158,29 @@ class CollectionMember(models.Model): def __str__(self): return '{} {}'.format(self.collection.uid, self.user) + def revoke(self): + with transaction.atomic(): + CollectionMemberRemoved.objects.update_or_create( + collection=self.collection, user=self.user, + defaults={ + 'stoken': Stoken.objects.create(), + }, + ) + + self.delete() + + +class CollectionMemberRemoved(models.Model): + stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT, null=True) + collection = models.ForeignKey(Collection, related_name='removed_members', on_delete=models.CASCADE) + user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + + class Meta: + unique_together = ('user', 'collection') + + def __str__(self): + return '{} {}'.format(self.collection.uid, self.user) + class CollectionInvitation(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 7079742..c5734d2 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -247,11 +247,10 @@ class CollectionMemberSerializer(serializers.ModelSerializer): slug_field=User.USERNAME_FIELD, queryset=User.objects ) - encryptionKey = BinaryBase64Field() class Meta: model = models.CollectionMember - fields = ('username', 'encryptionKey', 'accessLevel') + fields = ('username', 'accessLevel') def create(self, validated_data): raise NotImplementedError() @@ -324,6 +323,9 @@ class InvitationAcceptSerializer(serializers.Serializer): encryptionKey=encryption_key, ) + models.CollectionMemberRemoved.objects.filter( + user=invitation.user, collection=invitation.collection).delete() + invitation.delete() return member diff --git a/django_etesync/views.py b/django_etesync/views.py index 0f448fa..fd91aed 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -42,6 +42,7 @@ from .models import ( CollectionItem, CollectionItemRevision, CollectionMember, + CollectionMemberRemoved, CollectionInvitation, Stoken, UserInfo, @@ -181,6 +182,15 @@ class CollectionViewSet(BaseViewSet): 'data': serializer.data, 'stoken': new_stoken, } + + stoken_obj = self.get_stoken_obj(request) + if stoken_obj is not None: + # FIXME: honour limit? (the limit should be combined for data and this because of stoken) + remed = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) \ + .values_list('collection__uid', flat=True) + if len(remed) > 0: + ret['removedMemberships'] = [{'uid': x} for x in remed] + return Response(ret) @@ -417,7 +427,8 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): class CollectionMemberViewSet(BaseViewSet): allowed_methods = ['GET', 'PUT', 'DELETE'] - permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdmin, ) + our_base_permission_classes = BaseViewSet.permission_classes + permission_classes = our_base_permission_classes + (permissions.IsCollectionAdmin, ) queryset = CollectionMember.objects.all() serializer_class = CollectionMemberSerializer lookup_field = 'user__' + User.USERNAME_FIELD @@ -441,6 +452,21 @@ class CollectionMemberViewSet(BaseViewSet): def create(self, request): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) + # FIXME: block leaving if we are the last admins - should be deleted / assigned in this case depending if there + # are other memebers. + def perform_destroy(self, instance): + instance.revoke() + + @action_decorator(detail=False, methods=['POST'], permission_classes=our_base_permission_classes) + def leave(self, request, collection_uid=None): + collection_uid = self.kwargs['collection_uid'] + col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + + member = col.members.get(user=request.user) + self.perform_destroy(member) + + return Response({}) + class InvitationOutgoingViewSet(BaseViewSet): allowed_methods = ['GET', 'POST', 'PUT', 'DELETE'] From e159bf971bc0aea1aea38635a907cdf80ed9293f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 16:40:08 +0300 Subject: [PATCH 167/511] Collection/item viewsets: enforce access. --- django_etesync/permissions.py | 46 +++++++++++++++++++++++++++++++++++ django_etesync/views.py | 4 +-- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/django_etesync/permissions.py b/django_etesync/permissions.py index c371743..611977b 100644 --- a/django_etesync/permissions.py +++ b/django_etesync/permissions.py @@ -36,3 +36,49 @@ class IsCollectionAdmin(permissions.BasePermission): except Collection.DoesNotExist: # If the collection does not exist, we want to 404 later, not permission denied. return True + + +class IsCollectionAdminOrReadOnly(permissions.BasePermission): + """ + Custom permission to only allow owners of a collection to edit it + """ + message = 'Only collection admins can edit collections.' + code = 'admin_access_required' + + def has_permission(self, request, view): + collection_uid = view.kwargs.get('collection_uid', None) + + # Allow creating new collections + if collection_uid is None: + return True + + try: + collection = view.get_collection_queryset().get(uid=collection_uid) + if request.method in permissions.SAFE_METHODS: + return True + + return is_collection_admin(collection, request.user) + except Collection.DoesNotExist: + # If the collection does not exist, we want to 404 later, not permission denied. + return True + + +class HasWriteAccessOrReadOnly(permissions.BasePermission): + """ + Custom permission to restrict write + """ + message = 'You need write access to write to this collection' + code = 'no_write_access' + + def has_permission(self, request, view): + collection_uid = view.kwargs['collection_uid'] + try: + collection = view.get_collection_queryset().get(uid=collection_uid) + if request.method in permissions.SAFE_METHODS: + return True + else: + member = collection.members.get(user=request.user) + return member.accessLevel != AccessLevels.READ_ONLY + except Collection.DoesNotExist: + # If the collection does not exist, we want to 404 later, not permission denied. + return True diff --git a/django_etesync/views.py b/django_etesync/views.py index fd91aed..148081a 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -127,7 +127,7 @@ class BaseViewSet(viewsets.ModelViewSet): class CollectionViewSet(BaseViewSet): allowed_methods = ['GET', 'POST', 'DELETE'] - permission_classes = BaseViewSet.permission_classes + permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly, ) queryset = Collection.objects.all() serializer_class = CollectionSerializer lookup_field = 'uid' @@ -196,7 +196,7 @@ class CollectionViewSet(BaseViewSet): class CollectionItemViewSet(BaseViewSet): allowed_methods = ['GET', 'POST', 'PUT'] - permission_classes = BaseViewSet.permission_classes + permission_classes = BaseViewSet.permission_classes + (permissions.HasWriteAccessOrReadOnly, ) queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = 'uid' From f6960bb8cb198743830720825de72a1820ee28da Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 16:51:12 +0300 Subject: [PATCH 168/511] CollectionMember: fix collection list to return data in the right format. --- django_etesync/views.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index 148081a..38919a8 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -449,6 +449,16 @@ class CollectionMemberViewSet(BaseViewSet): return queryset.filter(collection=collection) + def list(self, request, collection_uid=None): + queryset = self.get_queryset() + serializer = self.get_serializer(queryset, many=True) + + ret = { + 'data': serializer.data, + } + + return Response(ret) + def create(self, request): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) From 6c31b8fb3033f1cd9f559c43cef0752b63f6a21b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 16:59:24 +0300 Subject: [PATCH 169/511] CollectionItemView: disallow normal item creation People should only use transaction/batch --- django_etesync/views.py | 18 ++---------------- 1 file changed, 2 insertions(+), 16 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 38919a8..1ba5fd1 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -222,22 +222,8 @@ class CollectionItemViewSet(BaseViewSet): return context def create(self, request, collection_uid=None): - collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - - # FIXME: change this to also support bulk update, or have another endpoint for that. - # See https://www.django-rest-framework.org/api-guide/serializers/#customizing-multiple-update - many = isinstance(request.data, list) - serializer = self.serializer_class(data=request.data, many=many) - if serializer.is_valid(): - try: - serializer.save(collection=collection_object) - except IntegrityError: - content = {'code': 'integrity_error'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - return Response({}, status=status.HTTP_201_CREATED) - - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + # We create using batch and transaction + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def destroy(self, request, collection_uid=None, uid=None): # We can't have destroy because we need to get data from the user (in the body) such as hmac. From 9f2140ffaca5d8a49e941ce1aa0f760e76e9a4df Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 17:00:33 +0300 Subject: [PATCH 170/511] Change serializer fetching to the more drf way of doing it. Also fix the ItemChunk serializer. --- django_etesync/views.py | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 1ba5fd1..31f0550 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -20,7 +20,6 @@ from django.contrib.auth import get_user_model from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError from django.db.models import Max, Q -from django.db.models.functions import Greatest from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -160,7 +159,7 @@ class CollectionViewSet(BaseViewSet): return Response({}) def create(self, request, *args, **kwargs): - serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) + serializer = self.get_serializer(data=request.data) if serializer.is_valid(): try: serializer.save(owner=self.request.user) @@ -176,7 +175,7 @@ class CollectionViewSet(BaseViewSet): queryset = self.get_queryset() queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) - serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) + serializer = self.get_serializer(queryset, many=True) ret = { 'data': serializer.data, @@ -239,7 +238,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = self.get_queryset() queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) - serializer = self.serializer_class(queryset, context=self.get_serializer_context(), many=True) + serializer = self.get_serializer(queryset, many=True) ret = { 'data': serializer.data, @@ -284,7 +283,7 @@ class CollectionItemViewSet(BaseViewSet): stoken = stoken_rev and stoken_rev.uid new_stoken = new_stoken or stoken - serializer = self.get_serializer_class()(queryset, context=self.get_serializer_context(), many=True) + serializer = self.get_serializer(queryset, many=True) ret = { 'data': serializer.data, @@ -304,8 +303,7 @@ class CollectionItemViewSet(BaseViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) items = request.data.get('items') - context = self.get_serializer_context() - serializer = self.get_serializer_class()(data=items, context=context, many=True) + serializer = self.get_serializer(data=items, many=True) if serializer.is_valid(): try: @@ -374,6 +372,9 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer_class = CollectionItemChunkSerializer lookup_field = 'uid' + def get_serializer_class(self): + return self.serializer_class + def get_collection_queryset(self, queryset=Collection.objects): user = self.request.user return queryset.filter(members__user=user) @@ -382,7 +383,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) - serializer = self.serializer_class(data=request.data) + serializer = self.get_serializer_class()(data=request.data) if serializer.is_valid(): try: serializer.save(item=col_it) @@ -484,7 +485,7 @@ class InvitationOutgoingViewSet(BaseViewSet): return queryset.filter(fromMember__user=self.request.user) def create(self, request, *args, **kwargs): - serializer = self.serializer_class(data=request.data, context=self.get_serializer_context()) + serializer = self.get_serializer(data=request.data) if serializer.is_valid(): collection_uid = serializer.validated_data.get('collection', {}).get('uid') From 89b47c67b7d98f2fd9ead10d308d0d46018b4b0f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 17:06:22 +0300 Subject: [PATCH 171/511] Removed redundant get_serializer_context. This is already provided by default in drf. --- django_etesync/views.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 31f0550..dfc0274 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -473,11 +473,6 @@ class InvitationOutgoingViewSet(BaseViewSet): lookup_field = 'uid' lookup_url_kwarg = 'invitation_uid' - def get_serializer_context(self): - context = super().get_serializer_context() - context.update({'request': self.request}) - return context - def get_queryset(self, queryset=None): if queryset is None: queryset = type(self).queryset From 64b947d455641c13454f33f18d347b622729a519 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 May 2020 17:14:38 +0300 Subject: [PATCH 172/511] Change invitations to also follow our list return type format. --- django_etesync/views.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index dfc0274..235d378 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -123,6 +123,17 @@ class BaseViewSet(viewsets.ModelViewSet): return queryset, new_stoken + # Change how our list works by default + def list(self, request, collection_uid=None): + queryset = self.get_queryset() + serializer = self.get_serializer(queryset, many=True) + + ret = { + 'data': serializer.data, + } + + return Response(ret) + class CollectionViewSet(BaseViewSet): allowed_methods = ['GET', 'POST', 'DELETE'] From 9347682997de3ff9d024b948d966620532568f20 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 13:29:03 +0300 Subject: [PATCH 173/511] Collection update: support limiting vs not limiting based on stoken. --- django_etesync/serializers.py | 6 +----- django_etesync/views.py | 7 +++++++ 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index c5734d2..3b50c36 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -203,7 +203,7 @@ class CollectionSerializer(serializers.ModelSerializer): with transaction.atomic(): if etag is not None: - raise serializers.ValidationError('Stoken is not None') + raise serializers.ValidationError('etag is not None') instance.save() main_item = models.CollectionItem.objects.create( @@ -222,13 +222,9 @@ class CollectionSerializer(serializers.ModelSerializer): def update(self, instance, validated_data): """Function that's called when this serializer is meant to update an item""" - etag = validated_data.pop('etag') revision_data = validated_data.pop('content') with transaction.atomic(): - if etag != instance.etag: - raise serializers.ValidationError('Wrong etag. Expected {} got {}'.format(instance.etag, etag)) - main_item = instance.main_item # We don't have to use select_for_update here because the unique constraint on current guards against # the race condition. But it's a good idea because it'll lock and wait rather than fail. diff --git a/django_etesync/views.py b/django_etesync/views.py index 235d378..dff1096 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -163,6 +163,13 @@ class CollectionViewSet(BaseViewSet): def update(self, request, *args, **kwargs): instance = self.get_object() + + stoken = request.GET.get('stoken', None) + + if stoken is not None and stoken != instance.stoken: + content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + serializer = self.get_serializer(instance, data=request.data) serializer.is_valid(raise_exception=True) self.perform_update(serializer) From ddc43c638acb2da8042ada6216a00c3f8a1778ca Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 14:56:14 +0300 Subject: [PATCH 174/511] Requirements: remove unused requirements. --- requirements.in/base.txt | 4 ---- requirements.txt | 16 ++++------------ 2 files changed, 4 insertions(+), 16 deletions(-) diff --git a/requirements.in/base.txt b/requirements.in/base.txt index e6d6379..1ab0e9a 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -1,11 +1,7 @@ django -django-allauth django-anymail -django-appconf django-cors-headers -django-debug-toolbar django-fullurl -django-ipware djangorestframework drf-nested-routers psycopg2-binary diff --git a/requirements.txt b/requirements.txt index dcc37cf..51b4d35 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,26 +8,18 @@ asgiref==3.2.3 # via django certifi==2019.11.28 # via requests cffi==1.14.0 # via pynacl chardet==3.0.4 # via requests -defusedxml==0.6.0 # via python3-openid -django-allauth==0.41.0 # via -r requirements.in/base.txt django-anymail==7.0.0 # via -r requirements.in/base.txt -django-appconf==1.0.3 # via -r requirements.in/base.txt django-cors-headers==3.2.1 # via -r requirements.in/base.txt -django-debug-toolbar==2.2 # via -r requirements.in/base.txt django-fullurl==1.0 # via -r requirements.in/base.txt -django-ipware==2.1.0 # via -r requirements.in/base.txt -django==3.0.3 # via -r requirements.in/base.txt, django-allauth, django-anymail, django-appconf, django-cors-headers, django-debug-toolbar, django-fullurl, djangorestframework, drf-nested-routers +django==3.0.3 # via -r requirements.in/base.txt, django-anymail, django-cors-headers, django-fullurl, djangorestframework, drf-nested-routers djangorestframework==3.11.0 # via -r requirements.in/base.txt, drf-nested-routers drf-nested-routers==0.91 # via -r requirements.in/base.txt idna==2.8 # via requests -oauthlib==3.1.0 # via requests-oauthlib psycopg2-binary==2.8.4 # via -r requirements.in/base.txt pycparser==2.20 # via cffi pynacl==1.3.0 # via -r requirements.in/base.txt -python3-openid==3.1.0 # via django-allauth pytz==2019.3 # via django -requests-oauthlib==1.3.0 # via django-allauth -requests==2.22.0 # via django-allauth, django-anymail, requests-oauthlib -six==1.14.0 # via django-anymail, django-appconf, pynacl -sqlparse==0.3.0 # via django, django-debug-toolbar +requests==2.22.0 # via django-anymail +six==1.14.0 # via django-anymail, pynacl +sqlparse==0.3.0 # via django urllib3==1.25.8 # via requests From 5b2040fda316f962bb39c82d57c0bd37ba85afc7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 16:05:19 +0300 Subject: [PATCH 175/511] Fix running with postgres: convert memoryview to bytes for nacl. --- django_etesync/views.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index dff1096..7864c6a 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -601,7 +601,7 @@ class AuthenticationViewSet(viewsets.ViewSet): if serializer.is_valid(): user = self.get_login_user(serializer) - salt = user.userinfo.salt + salt = bytes(user.userinfo.salt) enc_key = self.get_encryption_key(salt) box = nacl.secret.SecretBox(enc_key) @@ -637,7 +637,7 @@ class AuthenticationViewSet(viewsets.ViewSet): host = serializer.validated_data['host'] challenge = serializer.validated_data['challenge'] - salt = user.userinfo.salt + salt = bytes(user.userinfo.salt) enc_key = self.get_encryption_key(salt) box = nacl.secret.SecretBox(enc_key) @@ -654,7 +654,7 @@ class AuthenticationViewSet(viewsets.ViewSet): content = {'code': 'wrong_host', 'detail': detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) - verify_key = nacl.signing.VerifyKey(user.userinfo.loginPubkey, encoder=nacl.encoding.RawEncoder) + verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) verify_key.verify(response_raw, signature) data = self.login_response_data(user) From 40db4e14b0f1663f2833839b71769811232960cb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 16:05:46 +0300 Subject: [PATCH 176/511] Signup: rename the UserQuerySerializer to Signup. --- django_etesync/serializers.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 3b50c36..f117b31 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -339,12 +339,6 @@ class UserSerializer(serializers.ModelSerializer): fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, 'pubkey', 'encryptedSeckey') -class UserQuerySerializer(serializers.ModelSerializer): - class Meta: - model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) - - class UserInfoPubkeySerializer(serializers.ModelSerializer): pubkey = BinaryBase64Field() @@ -353,8 +347,14 @@ class UserInfoPubkeySerializer(serializers.ModelSerializer): fields = ('pubkey', ) +class UserSignupSerializer(serializers.ModelSerializer): + class Meta: + model = User + fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) + + class AuthenticationSignupSerializer(serializers.Serializer): - user = UserQuerySerializer(many=False) + user = UserSignupSerializer(many=False) salt = BinaryBase64Field() loginPubkey = BinaryBase64Field() pubkey = BinaryBase64Field() From 6051a5ae3a4250327b03ebc22cf562118b3fb218 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 16:06:59 +0300 Subject: [PATCH 177/511] Signup: use the recommended drf style for validation. --- django_etesync/views.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 7864c6a..52d3531 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -573,13 +573,11 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST']) def signup(self, request): serializer = AuthenticationSignupSerializer(data=request.data) - if serializer.is_valid(): - user = serializer.save() - - data = self.login_response_data(user) - return Response(data, status=status.HTTP_201_CREATED) + serializer.is_valid(raise_exception=True) + user = serializer.save() - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + data = self.login_response_data(user) + return Response(data, status=status.HTTP_201_CREATED) def get_login_user(self, serializer): username = serializer.validated_data.get('username') From 215a2607008b92d8f5fc082fc3a36cc4daf97910 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 16:13:43 +0300 Subject: [PATCH 178/511] Login: use only the username (not email) for login. We may add support for email in the future. --- django_etesync/serializers.py | 8 +------- django_etesync/views.py | 20 +++++++------------- 2 files changed, 8 insertions(+), 20 deletions(-) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index f117b31..337f695 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -381,13 +381,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): class AuthenticationLoginChallengeSerializer(serializers.Serializer): - username = serializers.CharField(required=False) - email = serializers.EmailField(required=False) - - def validate(self, data): - if not data.get('email') and not data.get('username'): - raise serializers.ValidationError('Either email or username must be set') - return data + username = serializers.CharField(required=True) def create(self, validated_data): raise NotImplementedError() diff --git a/django_etesync/views.py b/django_etesync/views.py index 52d3531..ac7a007 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -579,17 +579,9 @@ class AuthenticationViewSet(viewsets.ViewSet): data = self.login_response_data(user) return Response(data, status=status.HTTP_201_CREATED) - def get_login_user(self, serializer): - username = serializer.validated_data.get('username') - email = serializer.validated_data.get('email') - if username: - kwargs = {User.USERNAME_FIELD: username} - user = get_object_or_404(self.get_queryset(), **kwargs) - elif email: - kwargs = {User.EMAIL_FIELD: email} - user = get_object_or_404(self.get_queryset(), **kwargs) - - return user + def get_login_user(self, username): + kwargs = {User.USERNAME_FIELD: username} + return get_object_or_404(self.get_queryset(), **kwargs) @action_decorator(detail=False, methods=['POST']) def login_challenge(self, request): @@ -597,7 +589,8 @@ class AuthenticationViewSet(viewsets.ViewSet): serializer = AuthenticationLoginChallengeSerializer(data=request.data) if serializer.is_valid(): - user = self.get_login_user(serializer) + username = serializer.validated_data.get('username') + user = self.get_login_user(username) salt = bytes(user.userinfo.salt) enc_key = self.get_encryption_key(salt) @@ -631,7 +624,8 @@ class AuthenticationViewSet(viewsets.ViewSet): serializer = AuthenticationLoginInnerSerializer(data=response, context={'host': request.get_host()}) if serializer.is_valid(): - user = self.get_login_user(serializer) + username = serializer.validated_data.get('username') + user = self.get_login_user(username) host = serializer.validated_data['host'] challenge = serializer.validated_data['challenge'] From 15cd41db839833118dbbd6ec5ca5b91a8f61685d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 16:28:54 +0300 Subject: [PATCH 179/511] login: gracefully handle bad login attempts. --- django_etesync/views.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index ac7a007..e73d567 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -647,7 +647,11 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) - verify_key.verify(response_raw, signature) + + try: + verify_key.verify(response_raw, signature) + except nacl.exceptions.BadSignatureError: + return Response({'code': 'login_bad_signature'}, status=status.HTTP_400_BAD_REQUEST) data = self.login_response_data(user) return Response(data, status=status.HTTP_200_OK) From c2337f244d6089d0b7f983c565b759d53f4a7556 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 16:53:33 +0300 Subject: [PATCH 180/511] Signup: fix signup for users without user info. --- django_etesync/serializers.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 337f695..b3b83a0 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -15,6 +15,7 @@ import base64 from django.core.files.base import ContentFile +from django.core import exceptions as django_exceptions from django.contrib.auth import get_user_model from django.db import transaction from rest_framework import serializers @@ -351,6 +352,9 @@ class UserSignupSerializer(serializers.ModelSerializer): class Meta: model = User fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) + extra_kwargs = { + 'username': {'validators': []}, # We specifically validate in SignupSerializer + } class AuthenticationSignupSerializer(serializers.Serializer): @@ -370,6 +374,11 @@ class AuthenticationSignupSerializer(serializers.Serializer): raise serializers.ValidationError('User already exists') instance.set_unusable_password() + + try: + instance.clean_fields() + except django_exceptions.ValidationError as e: + raise serializers.ValidationError(e) # FIXME: send email verification models.UserInfo.objects.create(**validated_data, owner=instance) From 1bd4c5be52a4011adaa08b1befe93c430bf8df8d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 31 May 2020 18:26:21 +0300 Subject: [PATCH 181/511] Send the login signal on login. --- django_etesync/views.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index e73d567..ece894b 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -16,7 +16,7 @@ import json from functools import reduce from django.conf import settings -from django.contrib.auth import get_user_model +from django.contrib.auth import get_user_model, user_logged_in from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError from django.db.models import Max, Q @@ -654,6 +654,9 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response({'code': 'login_bad_signature'}, status=status.HTTP_400_BAD_REQUEST) data = self.login_response_data(user) + + user_logged_in.send(sender=user.__class__, request=request, user=user) + return Response(data, status=status.HTTP_200_OK) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) From 7842bd4d9cf06a0351f7a00a82b9e66eaced0b96 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 1 Jun 2020 12:45:06 +0300 Subject: [PATCH 182/511] CollectionItem list: don't return the main item. --- django_etesync/views.py | 1 + 1 file changed, 1 insertion(+) diff --git a/django_etesync/views.py b/django_etesync/views.py index ece894b..c41d0c7 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -227,6 +227,7 @@ class CollectionItemViewSet(BaseViewSet): raise Http404("Collection does not exist") # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = type(self).queryset.filter(collection__pk=collection.pk, + uid__isnull=False, revisions__current=True, revisions__deleted=False) From ad184f0ac3ae8d89bc70f00735ee86a5ff08443b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 2 Jun 2020 18:56:23 +0300 Subject: [PATCH 183/511] Rename encryptedSeckey to encryptedContent. --- .../migrations/0014_auto_20200602_1558.py | 18 ++++++++++++++++++ django_etesync/models.py | 2 +- django_etesync/serializers.py | 6 +++--- 3 files changed, 22 insertions(+), 4 deletions(-) create mode 100644 django_etesync/migrations/0014_auto_20200602_1558.py diff --git a/django_etesync/migrations/0014_auto_20200602_1558.py b/django_etesync/migrations/0014_auto_20200602_1558.py new file mode 100644 index 0000000..e226360 --- /dev/null +++ b/django_etesync/migrations/0014_auto_20200602_1558.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-06-02 15:58 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0013_collectionmemberremoved'), + ] + + operations = [ + migrations.RenameField( + model_name='userinfo', + old_name='encryptedSeckey', + new_name='encryptedContent', + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 7fbea61..18c14a9 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -214,7 +214,7 @@ class UserInfo(models.Model): version = models.PositiveSmallIntegerField(default=1) loginPubkey = models.BinaryField(editable=True, blank=False, null=False) pubkey = models.BinaryField(editable=True, blank=False, null=False) - encryptedSeckey = models.BinaryField(editable=True, blank=False, null=False) + encryptedContent = models.BinaryField(editable=True, blank=False, null=False) salt = models.BinaryField(editable=True, blank=False, null=False) def __str__(self): diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index b3b83a0..5925e46 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -333,11 +333,11 @@ class InvitationAcceptSerializer(serializers.Serializer): class UserSerializer(serializers.ModelSerializer): pubkey = BinaryBase64Field(source='userinfo.pubkey') - encryptedSeckey = BinaryBase64Field(source='userinfo.encryptedSeckey') + encryptedContent = BinaryBase64Field(source='userinfo.encryptedContent') class Meta: model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, 'pubkey', 'encryptedSeckey') + fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, 'pubkey', 'encryptedContent') class UserInfoPubkeySerializer(serializers.ModelSerializer): @@ -362,7 +362,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): salt = BinaryBase64Field() loginPubkey = BinaryBase64Field() pubkey = BinaryBase64Field() - encryptedSeckey = BinaryBase64Field() + encryptedContent = BinaryBase64Field() def create(self, validated_data): """Function that's called when this serializer creates an item""" From 9cc68291df7d693c16458ab11318cbb7624e52c7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 3 Jun 2020 14:21:52 +0300 Subject: [PATCH 184/511] Authentication classes: add permissions to logout. --- django_etesync/views.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index c41d0c7..3233b7b 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -553,6 +553,7 @@ class InvitationIncomingViewSet(BaseViewSet): class AuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] + authentication_classes = BaseViewSet.authentication_classes def get_encryption_key(self, salt): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) @@ -662,7 +663,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) def logout(self, request): # FIXME: expire the token - we need better token handling - using knox? Something else? return Response({}, status=status.HTTP_200_OK) From cc23d516a0c8aa64c522fded50b32169976bcbb7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 3 Jun 2020 14:35:44 +0300 Subject: [PATCH 185/511] Add an endpoint to change password. --- django_etesync/serializers.py | 20 ++++++++++++++++++++ django_etesync/views.py | 9 +++++++++ 2 files changed, 29 insertions(+) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 5925e46..178c914 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -419,3 +419,23 @@ class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer) def update(self, instance, validated_data): raise NotImplementedError() + + +class AuthenticationChangePasswordSerializer(serializers.ModelSerializer): + loginPubkey = BinaryBase64Field() + encryptedContent = BinaryBase64Field() + + class Meta: + model = models.UserInfo + fields = ('loginPubkey', 'encryptedContent') + + def create(self, validated_data): + raise NotImplementedError() + + def update(self, instance, validated_data): + with transaction.atomic(): + instance.loginPubkey = validated_data.pop('loginPubkey') + instance.encryptedContent = validated_data.pop('encryptedContent') + instance.save() + + return instance diff --git a/django_etesync/views.py b/django_etesync/views.py index 3233b7b..71ae93f 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -48,6 +48,7 @@ from .models import ( ) from .serializers import ( b64encode, + AuthenticationChangePasswordSerializer, AuthenticationSignupSerializer, AuthenticationLoginChallengeSerializer, AuthenticationLoginSerializer, @@ -668,6 +669,14 @@ class AuthenticationViewSet(viewsets.ViewSet): # FIXME: expire the token - we need better token handling - using knox? Something else? return Response({}, status=status.HTTP_200_OK) + @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) + def change_password(self, request): + serializer = AuthenticationChangePasswordSerializer(request.user.userinfo, data=request.data) + serializer.is_valid(raise_exception=True) + serializer.save() + + return Response(status=status.HTTP_200_OK) + class TestAuthenticationViewSet(viewsets.ViewSet): authentication_classes = BaseViewSet.authentication_classes From c00c208199e336a30375c8d9896327375fa61f44 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 3 Jun 2020 15:49:38 +0300 Subject: [PATCH 186/511] Change to our own token authentication. --- django_etesync/token_auth/__init__.py | 0 django_etesync/token_auth/admin.py | 0 django_etesync/token_auth/apps.py | 5 ++ django_etesync/token_auth/authentication.py | 46 +++++++++++++++++++ .../token_auth/migrations/0001_initial.py | 28 +++++++++++ .../token_auth/migrations/__init__.py | 0 django_etesync/token_auth/models.py | 26 +++++++++++ django_etesync/views.py | 12 +++-- 8 files changed, 112 insertions(+), 5 deletions(-) create mode 100644 django_etesync/token_auth/__init__.py create mode 100644 django_etesync/token_auth/admin.py create mode 100644 django_etesync/token_auth/apps.py create mode 100644 django_etesync/token_auth/authentication.py create mode 100644 django_etesync/token_auth/migrations/0001_initial.py create mode 100644 django_etesync/token_auth/migrations/__init__.py create mode 100644 django_etesync/token_auth/models.py diff --git a/django_etesync/token_auth/__init__.py b/django_etesync/token_auth/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etesync/token_auth/admin.py b/django_etesync/token_auth/admin.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etesync/token_auth/apps.py b/django_etesync/token_auth/apps.py new file mode 100644 index 0000000..dc793f2 --- /dev/null +++ b/django_etesync/token_auth/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class TokenAuthConfig(AppConfig): + name = 'django_etesync.token_auth' diff --git a/django_etesync/token_auth/authentication.py b/django_etesync/token_auth/authentication.py new file mode 100644 index 0000000..432c8cf --- /dev/null +++ b/django_etesync/token_auth/authentication.py @@ -0,0 +1,46 @@ +from django.utils import timezone +from django.utils.translation import gettext_lazy as _ + +from rest_framework import exceptions +from rest_framework.authentication import TokenAuthentication as DRFTokenAuthentication + +from .models import AuthToken, get_default_expiry + + +AUTO_REFRESH = True +MIN_REFRESH_INTERVAL = 60 + + +class TokenAuthentication(DRFTokenAuthentication): + keyword = 'Token' + model = AuthToken + + def authenticate_credentials(self, key): + msg = _('Invalid token.') + model = self.get_model() + try: + token = model.objects.select_related('user').get(key=key) + except model.DoesNotExist: + raise exceptions.AuthenticationFailed(msg) + + if not token.user.is_active: + raise exceptions.AuthenticationFailed(_('User inactive or deleted.')) + + if token.expiry is not None: + if token.expiry < timezone.now(): + token.delete() + raise exceptions.AuthenticationFailed(msg) + + if AUTO_REFRESH: + self.renew_token(token) + + return (token.user, token) + + def renew_token(self, auth_token): + current_expiry = auth_token.expiry + new_expiry = get_default_expiry() + # Throttle refreshing of token to avoid db writes + delta = (new_expiry - current_expiry).total_seconds() + if delta > MIN_REFRESH_INTERVAL: + auth_token.expiry = new_expiry + auth_token.save(update_fields=('expiry',)) diff --git a/django_etesync/token_auth/migrations/0001_initial.py b/django_etesync/token_auth/migrations/0001_initial.py new file mode 100644 index 0000000..f2024e3 --- /dev/null +++ b/django_etesync/token_auth/migrations/0001_initial.py @@ -0,0 +1,28 @@ +# Generated by Django 3.0.3 on 2020-06-03 12:49 + +from django.conf import settings +from django.db import migrations, models +import django.db.models.deletion +from django_etesync.token_auth import models as token_auth_models + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.CreateModel( + name='AuthToken', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('key', models.CharField(db_index=True, default=token_auth_models.generate_key, max_length=40, unique=True)), + ('created', models.DateTimeField(auto_now_add=True)), + ('expiry', models.DateTimeField(blank=True, default=token_auth_models.get_default_expiry, null=True)), + ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='auth_token_set', to=settings.AUTH_USER_MODEL)), + ], + ), + ] diff --git a/django_etesync/token_auth/migrations/__init__.py b/django_etesync/token_auth/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etesync/token_auth/models.py b/django_etesync/token_auth/models.py new file mode 100644 index 0000000..9ac0955 --- /dev/null +++ b/django_etesync/token_auth/models.py @@ -0,0 +1,26 @@ +from django.contrib.auth import get_user_model +from django.db import models +from django.utils import timezone +from django.utils.crypto import get_random_string + +User = get_user_model() + + +def generate_key(): + return get_random_string(40) + + +def get_default_expiry(): + return timezone.now() + timezone.timedelta(days=14) + + +class AuthToken(models.Model): + + key = models.CharField(max_length=40, unique=True, db_index=True, default=generate_key) + user = models.ForeignKey(User, null=False, blank=False, + related_name='auth_token_set', on_delete=models.CASCADE) + created = models.DateTimeField(auto_now_add=True) + expiry = models.DateTimeField(null=True, blank=True, default=get_default_expiry) + + def __str__(self): + return '{}: {}'.format(self.key, self.user) diff --git a/django_etesync/views.py b/django_etesync/views.py index 71ae93f..88ae7c4 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -16,7 +16,7 @@ import json from functools import reduce from django.conf import settings -from django.contrib.auth import get_user_model, user_logged_in +from django.contrib.auth import get_user_model, user_logged_in, user_logged_out from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError from django.db.models import Max, Q @@ -28,13 +28,14 @@ from rest_framework import viewsets from rest_framework import parsers from rest_framework.decorators import action as action_decorator from rest_framework.response import Response -from rest_framework.authtoken.models import Token import nacl.encoding import nacl.signing import nacl.secret import nacl.hash +from .token_auth.models import AuthToken + from . import app_settings, permissions from .models import ( Collection, @@ -566,7 +567,7 @@ class AuthenticationViewSet(viewsets.ViewSet): def login_response_data(self, user): return { - 'token': Token.objects.get_or_create(user=user)[0].key, + 'token': AuthToken.objects.create(user=user).key, 'user': UserSerializer(user).data, } @@ -666,8 +667,9 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) def logout(self, request): - # FIXME: expire the token - we need better token handling - using knox? Something else? - return Response({}, status=status.HTTP_200_OK) + request._auth.delete() + user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) + return Response(status=status.HTTP_204_NO_CONTENT) @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) def change_password(self, request): From 29145f22156f30e74e988b8b3efab89db4f510b7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 3 Jun 2020 16:19:07 +0300 Subject: [PATCH 187/511] Logout: don't use internal auth accessor. --- django_etesync/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 88ae7c4..245120b 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -667,7 +667,7 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) def logout(self, request): - request._auth.delete() + request.auth.delete() user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) return Response(status=status.HTTP_204_NO_CONTENT) From 119479d22b6a63fc0b647b552b497016f106d06f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 3 Jun 2020 17:22:10 +0300 Subject: [PATCH 188/511] Test reset: allow anyone to reset test users and fully init accounts. --- django_etesync/views.py | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index 245120b..f23971d 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -681,8 +681,6 @@ class AuthenticationViewSet(viewsets.ViewSet): class TestAuthenticationViewSet(viewsets.ViewSet): - authentication_classes = BaseViewSet.authentication_classes - permission_classes = BaseViewSet.permission_classes allowed_methods = ['POST'] def list(self, request): @@ -694,13 +692,22 @@ class TestAuthenticationViewSet(viewsets.ViewSet): if not settings.DEBUG: return HttpResponseBadRequest("Only allowed in debug mode.") - # Only allow local users, for extra safety - if not getattr(request.user, User.EMAIL_FIELD).endswith('@localhost'): + user = get_object_or_404(User.objects.all(), username=request.data.get('user').get('username')) + + # Only allow test users for extra safety + if not getattr(user, User.USERNAME_FIELD).startswith('test_user'): return HttpResponseBadRequest("Endpoint not allowed for user.") + if hasattr(user, 'userinfo'): + user.userinfo.delete() + + serializer = AuthenticationSignupSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + serializer.save() + # Delete all of the journal data for this user for a clear test env - request.user.collection_set.all().delete() - request.user.incoming_invitations.all().delete() + user.collection_set.all().delete() + user.incoming_invitations.all().delete() # FIXME: also delete chunk files!!! From e062fcd4298e6f0b995f6759cc1a32d6504691e1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 4 Jun 2020 15:23:10 +0300 Subject: [PATCH 189/511] Revision: add salt. --- .../0015_collectionitemrevision_salt.py | 18 ++++++++++++++++++ django_etesync/models.py | 1 + django_etesync/serializers.py | 3 ++- 3 files changed, 21 insertions(+), 1 deletion(-) create mode 100644 django_etesync/migrations/0015_collectionitemrevision_salt.py diff --git a/django_etesync/migrations/0015_collectionitemrevision_salt.py b/django_etesync/migrations/0015_collectionitemrevision_salt.py new file mode 100644 index 0000000..f5553c9 --- /dev/null +++ b/django_etesync/migrations/0015_collectionitemrevision_salt.py @@ -0,0 +1,18 @@ +# Generated by Django 3.0.3 on 2020-06-04 12:18 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etesync', '0014_auto_20200602_1558'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemrevision', + name='salt', + field=models.BinaryField(default=b'', editable=True), + ), + ] diff --git a/django_etesync/models.py b/django_etesync/models.py index 18c14a9..53239e7 100644 --- a/django_etesync/models.py +++ b/django_etesync/models.py @@ -115,6 +115,7 @@ class CollectionItemRevision(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) uid = models.CharField(db_index=True, unique=True, blank=False, null=False, max_length=43, validators=[Base64Url256BitlValidator]) + salt = models.BinaryField(editable=True, blank=False, null=False, default=b'') item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 178c914..1f5d3c2 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -106,11 +106,12 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): queryset=models.RevisionChunkRelation.objects.all(), many=True ) + salt = BinaryBase64Field() meta = BinaryBase64Field() class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'meta', 'uid', 'deleted') + fields = ('chunks', 'meta', 'uid', 'salt', 'deleted') class CollectionItemSerializer(serializers.ModelSerializer): From 653341115f1cb1e2e9515da1b8cea6e9e4e8e0d7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 4 Jun 2020 16:52:56 +0300 Subject: [PATCH 190/511] Chunks: add stricter validation. --- django_etesync/serializers.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/django_etesync/serializers.py b/django_etesync/serializers.py index 1f5d3c2..c940d6e 100644 --- a/django_etesync/serializers.py +++ b/django_etesync/serializers.py @@ -91,6 +91,8 @@ class ChunksField(serializers.RelatedField): return (obj.uid, ) def to_internal_value(self, data): + if data[0] is None or data[1] is None: + raise serializers.ValidationError('null is not allowed') return (data[0], b64decode(data[1])) From 23b2bb3c0a0be5c24f3f5b1555fb733d8e161df0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 11 Jun 2020 11:17:01 +0300 Subject: [PATCH 191/511] Batch: refactor code and allow passing deps to check against. --- django_etesync/views.py | 93 +++++++++++++++-------------------------- 1 file changed, 34 insertions(+), 59 deletions(-) diff --git a/django_etesync/views.py b/django_etesync/views.py index f23971d..1d29ae1 100644 --- a/django_etesync/views.py +++ b/django_etesync/views.py @@ -316,73 +316,48 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST']) def batch(self, request, collection_uid=None): - stoken = request.GET.get('stoken', None) - collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - - if stoken is not None and stoken != collection_object.stoken: - content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - items = request.data.get('items') - serializer = self.get_serializer(data=items, many=True) - - if serializer.is_valid(): - try: - with transaction.atomic(): - items = serializer.save(collection=collection_object) - except IntegrityError: - # FIXME: should return the items with a bad token (including deps) so we don't have to fetch them after - content = {'code': 'integrity_error'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - ret = { - } - return Response(ret, status=status.HTTP_200_OK) - - return Response( - { - "items": serializer.errors, - }, - status=status.HTTP_400_BAD_REQUEST) + return self.transaction(request, collection_uid, validate_etag=False) @action_decorator(detail=False, methods=['POST']) - def transaction(self, request, collection_uid=None): + def transaction(self, request, collection_uid=None, validate_etag=True): stoken = request.GET.get('stoken', None) - collection_object = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - - if stoken is not None and stoken != collection_object.stoken: - content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) + with transaction.atomic(): # We need this for locking on the collection object + collection_object = get_object_or_404( + self.get_collection_queryset(Collection.objects).select_for_update(), # Lock writes on the collection + uid=collection_uid) - items = request.data.get('items') - deps = request.data.get('deps', None) - # FIXME: It should just be one serializer - context = self.get_serializer_context() - context.update({'validate_etag': True}) - serializer = self.get_serializer_class()(data=items, context=context, many=True) - deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) + if stoken is not None and stoken != collection_object.stoken: + content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) - ser_valid = serializer.is_valid() - deps_ser_valid = (deps is None or deps_serializer.is_valid()) - if ser_valid and deps_ser_valid: - try: - with transaction.atomic(): + items = request.data.get('items') + deps = request.data.get('deps', None) + # FIXME: It should just be one serializer + context = self.get_serializer_context() + context.update({'validate_etag': validate_etag}) + serializer = self.get_serializer_class()(data=items, context=context, many=True) + deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) + + ser_valid = serializer.is_valid() + deps_ser_valid = (deps is None or deps_serializer.is_valid()) + if ser_valid and deps_ser_valid: + try: items = serializer.save(collection=collection_object) - except IntegrityError: - # FIXME: should return the items with a bad token (including deps) so we don't have to fetch them after - content = {'code': 'integrity_error'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) + except IntegrityError: + # FIXME: return the items with a bad token (including deps) so we don't have to fetch them after + content = {'code': 'integrity_error'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) - ret = { - } - return Response(ret, status=status.HTTP_200_OK) + ret = { + } + return Response(ret, status=status.HTTP_200_OK) - return Response( - { - "items": serializer.errors, - "deps": deps_serializer.errors if deps is not None else [], - }, - status=status.HTTP_400_BAD_REQUEST) + return Response( + { + "items": serializer.errors, + "deps": deps_serializer.errors if deps is not None else [], + }, + status=status.HTTP_400_BAD_REQUEST) class CollectionItemChunkViewSet(viewsets.ViewSet): From d1017aac761f06c2e2f7a36c2768da0bdd0edc7a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 16 Jun 2020 17:12:44 +0300 Subject: [PATCH 192/511] Rename django_etesync to django_etebase. --- {django_etesync => django_etebase}/__init__.py | 0 {django_etesync => django_etebase}/admin.py | 0 .../app_settings.py | 2 +- django_etebase/apps.py | 5 +++++ .../migrations/0001_initial.py | 16 ++++++++-------- .../migrations/0002_userinfo.py | 2 +- .../migrations/0003_collectioninvitation.py | 4 ++-- .../0004_collectioninvitation_version.py | 2 +- .../migrations/0005_auto_20200526_1021.py | 2 +- .../migrations/0006_auto_20200526_1040.py | 2 +- .../migrations/0007_auto_20200526_1336.py | 2 +- .../migrations/0008_auto_20200526_1535.py | 8 ++++---- .../migrations/0009_auto_20200526_1535.py | 6 +++--- .../migrations/0010_auto_20200526_1539.py | 4 ++-- .../migrations/0011_collectionmember_stoken.py | 4 ++-- .../migrations/0012_auto_20200527_0743.py | 6 +++--- .../migrations/0013_collectionmemberremoved.py | 6 +++--- .../migrations/0014_auto_20200602_1558.py | 2 +- .../0015_collectionitemrevision_salt.py | 2 +- .../migrations/__init__.py | 0 {django_etesync => django_etebase}/models.py | 0 .../permissions.py | 2 +- .../serializers.py | 0 {django_etesync => django_etebase}/tests.py | 0 .../token_auth/__init__.py | 0 .../token_auth/admin.py | 0 .../token_auth/apps.py | 2 +- .../token_auth/authentication.py | 0 .../token_auth/migrations/0001_initial.py | 2 +- .../token_auth/migrations/__init__.py | 0 .../token_auth/models.py | 0 {django_etesync => django_etebase}/views.py | 2 +- django_etesync/apps.py | 5 ----- 33 files changed, 44 insertions(+), 44 deletions(-) rename {django_etesync => django_etebase}/__init__.py (100%) rename {django_etesync => django_etebase}/admin.py (100%) rename {django_etesync => django_etebase}/app_settings.py (97%) create mode 100644 django_etebase/apps.py rename {django_etesync => django_etebase}/migrations/0001_initial.py (93%) rename {django_etesync => django_etebase}/migrations/0002_userinfo.py (94%) rename {django_etesync => django_etebase}/migrations/0003_collectioninvitation.py (92%) rename {django_etesync => django_etebase}/migrations/0004_collectioninvitation_version.py (86%) rename {django_etesync => django_etebase}/migrations/0005_auto_20200526_1021.py (83%) rename {django_etesync => django_etebase}/migrations/0006_auto_20200526_1040.py (91%) rename {django_etesync => django_etebase}/migrations/0007_auto_20200526_1336.py (96%) rename {django_etesync => django_etebase}/migrations/0008_auto_20200526_1535.py (80%) rename {django_etesync => django_etebase}/migrations/0009_auto_20200526_1535.py (69%) rename {django_etesync => django_etebase}/migrations/0010_auto_20200526_1539.py (79%) rename {django_etesync => django_etebase}/migrations/0011_collectionmember_stoken.py (77%) rename {django_etesync => django_etebase}/migrations/0012_auto_20200527_0743.py (68%) rename {django_etesync => django_etebase}/migrations/0013_collectionmemberremoved.py (86%) rename {django_etesync => django_etebase}/migrations/0014_auto_20200602_1558.py (84%) rename {django_etesync => django_etebase}/migrations/0015_collectionitemrevision_salt.py (86%) rename {django_etesync => django_etebase}/migrations/__init__.py (100%) rename {django_etesync => django_etebase}/models.py (100%) rename {django_etesync => django_etebase}/permissions.py (98%) rename {django_etesync => django_etebase}/serializers.py (100%) rename {django_etesync => django_etebase}/tests.py (100%) rename {django_etesync => django_etebase}/token_auth/__init__.py (100%) rename {django_etesync => django_etebase}/token_auth/admin.py (100%) rename {django_etesync => django_etebase}/token_auth/apps.py (64%) rename {django_etesync => django_etebase}/token_auth/authentication.py (100%) rename {django_etesync => django_etebase}/token_auth/migrations/0001_initial.py (94%) rename {django_etesync => django_etebase}/token_auth/migrations/__init__.py (100%) rename {django_etesync => django_etebase}/token_auth/models.py (100%) rename {django_etesync => django_etebase}/views.py (99%) delete mode 100644 django_etesync/apps.py diff --git a/django_etesync/__init__.py b/django_etebase/__init__.py similarity index 100% rename from django_etesync/__init__.py rename to django_etebase/__init__.py diff --git a/django_etesync/admin.py b/django_etebase/admin.py similarity index 100% rename from django_etesync/admin.py rename to django_etebase/admin.py diff --git a/django_etesync/app_settings.py b/django_etebase/app_settings.py similarity index 97% rename from django_etesync/app_settings.py rename to django_etebase/app_settings.py index 89b38f7..b1fb4c3 100644 --- a/django_etesync/app_settings.py +++ b/django_etebase/app_settings.py @@ -51,4 +51,4 @@ class AppSettings: return self._setting("CHALLENGE_VALID_SECONDS", 60) -app_settings = AppSettings('ETESYNC_') +app_settings = AppSettings('ETEBASE_') diff --git a/django_etebase/apps.py b/django_etebase/apps.py new file mode 100644 index 0000000..286a708 --- /dev/null +++ b/django_etebase/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class DjangoEtebaseConfig(AppConfig): + name = 'django_etebase' diff --git a/django_etesync/migrations/0001_initial.py b/django_etebase/migrations/0001_initial.py similarity index 93% rename from django_etesync/migrations/0001_initial.py rename to django_etebase/migrations/0001_initial.py index dc3a2ff..69a9a91 100644 --- a/django_etesync/migrations/0001_initial.py +++ b/django_etebase/migrations/0001_initial.py @@ -4,7 +4,7 @@ from django.conf import settings import django.core.validators from django.db import migrations, models import django.db.models.deletion -import django_etesync.models +import django_etebase.models class Migration(migrations.Migration): @@ -35,7 +35,7 @@ class Migration(migrations.Migration): ('uid', models.CharField(db_index=True, max_length=44, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')])), ('version', models.PositiveSmallIntegerField()), ('encryptionKey', models.BinaryField(editable=True, null=True)), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='items', to='django_etesync.Collection')), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='items', to='django_etebase.Collection')), ], options={ 'unique_together': {('uid', 'collection')}, @@ -46,8 +46,8 @@ class Migration(migrations.Migration): fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), - ('chunkFile', models.FileField(max_length=150, unique=True, upload_to=django_etesync.models.chunk_directory_path)), - ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etesync.CollectionItem')), + ('chunkFile', models.FileField(max_length=150, unique=True, upload_to=django_etebase.models.chunk_directory_path)), + ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etebase.CollectionItem')), ], ), migrations.CreateModel( @@ -58,7 +58,7 @@ class Migration(migrations.Migration): ('meta', models.BinaryField(editable=True)), ('current', models.BooleanField(db_index=True, default=True, null=True)), ('deleted', models.BooleanField(default=False)), - ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='django_etesync.CollectionItem')), + ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='django_etebase.CollectionItem')), ], options={ 'unique_together': {('item', 'current')}, @@ -68,8 +68,8 @@ class Migration(migrations.Migration): name='RevisionChunkRelation', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('chunk', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions_relation', to='django_etesync.CollectionItemChunk')), - ('revision', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks_relation', to='django_etesync.CollectionItemRevision')), + ('chunk', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions_relation', to='django_etebase.CollectionItemChunk')), + ('revision', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks_relation', to='django_etebase.CollectionItemRevision')), ], options={ 'ordering': ('id',), @@ -81,7 +81,7 @@ class Migration(migrations.Migration): ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('encryptionKey', models.BinaryField(editable=True)), ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='members', to='django_etesync.Collection')), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='members', to='django_etebase.Collection')), ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], options={ diff --git a/django_etesync/migrations/0002_userinfo.py b/django_etebase/migrations/0002_userinfo.py similarity index 94% rename from django_etesync/migrations/0002_userinfo.py rename to django_etebase/migrations/0002_userinfo.py index ad7018a..6da0bb8 100644 --- a/django_etesync/migrations/0002_userinfo.py +++ b/django_etebase/migrations/0002_userinfo.py @@ -9,7 +9,7 @@ class Migration(migrations.Migration): dependencies = [ ('myauth', '0001_initial'), - ('django_etesync', '0001_initial'), + ('django_etebase', '0001_initial'), ] operations = [ diff --git a/django_etesync/migrations/0003_collectioninvitation.py b/django_etebase/migrations/0003_collectioninvitation.py similarity index 92% rename from django_etesync/migrations/0003_collectioninvitation.py rename to django_etebase/migrations/0003_collectioninvitation.py index 3880a63..8fd2066 100644 --- a/django_etesync/migrations/0003_collectioninvitation.py +++ b/django_etebase/migrations/0003_collectioninvitation.py @@ -10,7 +10,7 @@ class Migration(migrations.Migration): dependencies = [ migrations.swappable_dependency(settings.AUTH_USER_MODEL), - ('django_etesync', '0002_userinfo'), + ('django_etebase', '0002_userinfo'), ] operations = [ @@ -21,7 +21,7 @@ class Migration(migrations.Migration): ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), ('signedEncryptionKey', models.BinaryField()), ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), - ('fromMember', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etesync.CollectionMember')), + ('fromMember', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etebase.CollectionMember')), ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='incoming_invitations', to=settings.AUTH_USER_MODEL)), ], options={ diff --git a/django_etesync/migrations/0004_collectioninvitation_version.py b/django_etebase/migrations/0004_collectioninvitation_version.py similarity index 86% rename from django_etesync/migrations/0004_collectioninvitation_version.py rename to django_etebase/migrations/0004_collectioninvitation_version.py index 3fbaed9..4052116 100644 --- a/django_etesync/migrations/0004_collectioninvitation_version.py +++ b/django_etebase/migrations/0004_collectioninvitation_version.py @@ -6,7 +6,7 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0003_collectioninvitation'), + ('django_etebase', '0003_collectioninvitation'), ] operations = [ diff --git a/django_etesync/migrations/0005_auto_20200526_1021.py b/django_etebase/migrations/0005_auto_20200526_1021.py similarity index 83% rename from django_etesync/migrations/0005_auto_20200526_1021.py rename to django_etebase/migrations/0005_auto_20200526_1021.py index 470556b..da0dc33 100644 --- a/django_etesync/migrations/0005_auto_20200526_1021.py +++ b/django_etebase/migrations/0005_auto_20200526_1021.py @@ -6,7 +6,7 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0004_collectioninvitation_version'), + ('django_etebase', '0004_collectioninvitation_version'), ] operations = [ diff --git a/django_etesync/migrations/0006_auto_20200526_1040.py b/django_etebase/migrations/0006_auto_20200526_1040.py similarity index 91% rename from django_etesync/migrations/0006_auto_20200526_1040.py rename to django_etebase/migrations/0006_auto_20200526_1040.py index 84e8fa3..b86a996 100644 --- a/django_etesync/migrations/0006_auto_20200526_1040.py +++ b/django_etebase/migrations/0006_auto_20200526_1040.py @@ -6,7 +6,7 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0005_auto_20200526_1021'), + ('django_etebase', '0005_auto_20200526_1021'), ] operations = [ diff --git a/django_etesync/migrations/0007_auto_20200526_1336.py b/django_etebase/migrations/0007_auto_20200526_1336.py similarity index 96% rename from django_etesync/migrations/0007_auto_20200526_1336.py rename to django_etebase/migrations/0007_auto_20200526_1336.py index 37e31ac..79978c7 100644 --- a/django_etesync/migrations/0007_auto_20200526_1336.py +++ b/django_etebase/migrations/0007_auto_20200526_1336.py @@ -7,7 +7,7 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0006_auto_20200526_1040'), + ('django_etebase', '0006_auto_20200526_1040'), ] operations = [ diff --git a/django_etesync/migrations/0008_auto_20200526_1535.py b/django_etebase/migrations/0008_auto_20200526_1535.py similarity index 80% rename from django_etesync/migrations/0008_auto_20200526_1535.py rename to django_etebase/migrations/0008_auto_20200526_1535.py index e544bdd..12656c0 100644 --- a/django_etesync/migrations/0008_auto_20200526_1535.py +++ b/django_etebase/migrations/0008_auto_20200526_1535.py @@ -3,13 +3,13 @@ import django.core.validators from django.db import migrations, models import django.db.models.deletion -import django_etesync.models +import django_etebase.models class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0007_auto_20200526_1336'), + ('django_etebase', '0007_auto_20200526_1336'), ] operations = [ @@ -17,12 +17,12 @@ class Migration(migrations.Migration): name='Stoken', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, default=django_etesync.models.generate_stoken_uid, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')])), + ('uid', models.CharField(db_index=True, default=django_etebase.models.generate_stoken_uid, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')])), ], ), migrations.AddField( model_name='collectionitemrevision', name='stoken', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken'), + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken'), ), ] diff --git a/django_etesync/migrations/0009_auto_20200526_1535.py b/django_etebase/migrations/0009_auto_20200526_1535.py similarity index 69% rename from django_etesync/migrations/0009_auto_20200526_1535.py rename to django_etebase/migrations/0009_auto_20200526_1535.py index 53100b3..a6ff498 100644 --- a/django_etesync/migrations/0009_auto_20200526_1535.py +++ b/django_etebase/migrations/0009_auto_20200526_1535.py @@ -4,8 +4,8 @@ from django.db import migrations def create_stokens(apps, schema_editor): - Stoken = apps.get_model('django_etesync', 'Stoken') - CollectionItemRevision = apps.get_model('django_etesync', 'CollectionItemRevision') + Stoken = apps.get_model('django_etebase', 'Stoken') + CollectionItemRevision = apps.get_model('django_etebase', 'CollectionItemRevision') for rev in CollectionItemRevision.objects.all(): rev.stoken = Stoken.objects.create() @@ -15,7 +15,7 @@ def create_stokens(apps, schema_editor): class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0008_auto_20200526_1535'), + ('django_etebase', '0008_auto_20200526_1535'), ] operations = [ diff --git a/django_etesync/migrations/0010_auto_20200526_1539.py b/django_etebase/migrations/0010_auto_20200526_1539.py similarity index 79% rename from django_etesync/migrations/0010_auto_20200526_1539.py rename to django_etebase/migrations/0010_auto_20200526_1539.py index c894fd2..7ef0eca 100644 --- a/django_etesync/migrations/0010_auto_20200526_1539.py +++ b/django_etebase/migrations/0010_auto_20200526_1539.py @@ -7,13 +7,13 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0009_auto_20200526_1535'), + ('django_etebase', '0009_auto_20200526_1535'), ] operations = [ migrations.AlterField( model_name='collectionitemrevision', name='stoken', - field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken'), + field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken'), ), ] diff --git a/django_etesync/migrations/0011_collectionmember_stoken.py b/django_etebase/migrations/0011_collectionmember_stoken.py similarity index 77% rename from django_etesync/migrations/0011_collectionmember_stoken.py rename to django_etebase/migrations/0011_collectionmember_stoken.py index 6b79bf6..bafaea7 100644 --- a/django_etesync/migrations/0011_collectionmember_stoken.py +++ b/django_etebase/migrations/0011_collectionmember_stoken.py @@ -7,13 +7,13 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0010_auto_20200526_1539'), + ('django_etebase', '0010_auto_20200526_1539'), ] operations = [ migrations.AddField( model_name='collectionmember', name='stoken', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken'), + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken'), ), ] diff --git a/django_etesync/migrations/0012_auto_20200527_0743.py b/django_etebase/migrations/0012_auto_20200527_0743.py similarity index 68% rename from django_etesync/migrations/0012_auto_20200527_0743.py rename to django_etebase/migrations/0012_auto_20200527_0743.py index 28b8745..ab6adbc 100644 --- a/django_etesync/migrations/0012_auto_20200527_0743.py +++ b/django_etebase/migrations/0012_auto_20200527_0743.py @@ -4,8 +4,8 @@ from django.db import migrations def create_stokens(apps, schema_editor): - Stoken = apps.get_model('django_etesync', 'Stoken') - CollectionMember = apps.get_model('django_etesync', 'CollectionMember') + Stoken = apps.get_model('django_etebase', 'Stoken') + CollectionMember = apps.get_model('django_etebase', 'CollectionMember') for member in CollectionMember.objects.all(): member.stoken = Stoken.objects.create() @@ -15,7 +15,7 @@ def create_stokens(apps, schema_editor): class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0011_collectionmember_stoken'), + ('django_etebase', '0011_collectionmember_stoken'), ] operations = [ diff --git a/django_etesync/migrations/0013_collectionmemberremoved.py b/django_etebase/migrations/0013_collectionmemberremoved.py similarity index 86% rename from django_etesync/migrations/0013_collectionmemberremoved.py rename to django_etebase/migrations/0013_collectionmemberremoved.py index e796232..2641c03 100644 --- a/django_etesync/migrations/0013_collectionmemberremoved.py +++ b/django_etebase/migrations/0013_collectionmemberremoved.py @@ -9,7 +9,7 @@ class Migration(migrations.Migration): dependencies = [ migrations.swappable_dependency(settings.AUTH_USER_MODEL), - ('django_etesync', '0012_auto_20200527_0743'), + ('django_etebase', '0012_auto_20200527_0743'), ] operations = [ @@ -17,8 +17,8 @@ class Migration(migrations.Migration): name='CollectionMemberRemoved', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='removed_members', to='django_etesync.Collection')), - ('stoken', models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etesync.Stoken')), + ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='removed_members', to='django_etebase.Collection')), + ('stoken', models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken')), ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], options={ diff --git a/django_etesync/migrations/0014_auto_20200602_1558.py b/django_etebase/migrations/0014_auto_20200602_1558.py similarity index 84% rename from django_etesync/migrations/0014_auto_20200602_1558.py rename to django_etebase/migrations/0014_auto_20200602_1558.py index e226360..d1a555d 100644 --- a/django_etesync/migrations/0014_auto_20200602_1558.py +++ b/django_etebase/migrations/0014_auto_20200602_1558.py @@ -6,7 +6,7 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0013_collectionmemberremoved'), + ('django_etebase', '0013_collectionmemberremoved'), ] operations = [ diff --git a/django_etesync/migrations/0015_collectionitemrevision_salt.py b/django_etebase/migrations/0015_collectionitemrevision_salt.py similarity index 86% rename from django_etesync/migrations/0015_collectionitemrevision_salt.py rename to django_etebase/migrations/0015_collectionitemrevision_salt.py index f5553c9..7f3dd71 100644 --- a/django_etesync/migrations/0015_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0015_collectionitemrevision_salt.py @@ -6,7 +6,7 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etesync', '0014_auto_20200602_1558'), + ('django_etebase', '0014_auto_20200602_1558'), ] operations = [ diff --git a/django_etesync/migrations/__init__.py b/django_etebase/migrations/__init__.py similarity index 100% rename from django_etesync/migrations/__init__.py rename to django_etebase/migrations/__init__.py diff --git a/django_etesync/models.py b/django_etebase/models.py similarity index 100% rename from django_etesync/models.py rename to django_etebase/models.py diff --git a/django_etesync/permissions.py b/django_etebase/permissions.py similarity index 98% rename from django_etesync/permissions.py rename to django_etebase/permissions.py index 611977b..a4217a8 100644 --- a/django_etesync/permissions.py +++ b/django_etebase/permissions.py @@ -13,7 +13,7 @@ # along with this program. If not, see . from rest_framework import permissions -from django_etesync.models import Collection, AccessLevels +from django_etebase.models import Collection, AccessLevels def is_collection_admin(collection, user): diff --git a/django_etesync/serializers.py b/django_etebase/serializers.py similarity index 100% rename from django_etesync/serializers.py rename to django_etebase/serializers.py diff --git a/django_etesync/tests.py b/django_etebase/tests.py similarity index 100% rename from django_etesync/tests.py rename to django_etebase/tests.py diff --git a/django_etesync/token_auth/__init__.py b/django_etebase/token_auth/__init__.py similarity index 100% rename from django_etesync/token_auth/__init__.py rename to django_etebase/token_auth/__init__.py diff --git a/django_etesync/token_auth/admin.py b/django_etebase/token_auth/admin.py similarity index 100% rename from django_etesync/token_auth/admin.py rename to django_etebase/token_auth/admin.py diff --git a/django_etesync/token_auth/apps.py b/django_etebase/token_auth/apps.py similarity index 64% rename from django_etesync/token_auth/apps.py rename to django_etebase/token_auth/apps.py index dc793f2..118b872 100644 --- a/django_etesync/token_auth/apps.py +++ b/django_etebase/token_auth/apps.py @@ -2,4 +2,4 @@ from django.apps import AppConfig class TokenAuthConfig(AppConfig): - name = 'django_etesync.token_auth' + name = 'django_etebase.token_auth' diff --git a/django_etesync/token_auth/authentication.py b/django_etebase/token_auth/authentication.py similarity index 100% rename from django_etesync/token_auth/authentication.py rename to django_etebase/token_auth/authentication.py diff --git a/django_etesync/token_auth/migrations/0001_initial.py b/django_etebase/token_auth/migrations/0001_initial.py similarity index 94% rename from django_etesync/token_auth/migrations/0001_initial.py rename to django_etebase/token_auth/migrations/0001_initial.py index f2024e3..5a47366 100644 --- a/django_etesync/token_auth/migrations/0001_initial.py +++ b/django_etebase/token_auth/migrations/0001_initial.py @@ -3,7 +3,7 @@ from django.conf import settings from django.db import migrations, models import django.db.models.deletion -from django_etesync.token_auth import models as token_auth_models +from django_etebase.token_auth import models as token_auth_models class Migration(migrations.Migration): diff --git a/django_etesync/token_auth/migrations/__init__.py b/django_etebase/token_auth/migrations/__init__.py similarity index 100% rename from django_etesync/token_auth/migrations/__init__.py rename to django_etebase/token_auth/migrations/__init__.py diff --git a/django_etesync/token_auth/models.py b/django_etebase/token_auth/models.py similarity index 100% rename from django_etesync/token_auth/models.py rename to django_etebase/token_auth/models.py diff --git a/django_etesync/views.py b/django_etebase/views.py similarity index 99% rename from django_etesync/views.py rename to django_etebase/views.py index 1d29ae1..59fcaa2 100644 --- a/django_etesync/views.py +++ b/django_etebase/views.py @@ -534,7 +534,7 @@ class AuthenticationViewSet(viewsets.ViewSet): def get_encryption_key(self, salt): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) - return nacl.hash.blake2b(b'', key=key, salt=salt[:nacl.hash.BLAKE2B_SALTBYTES], person=b'etesync-auth', + return nacl.hash.blake2b(b'', key=key, salt=salt[:nacl.hash.BLAKE2B_SALTBYTES], person=b'etebase-auth', encoder=nacl.encoding.RawEncoder) def get_queryset(self): diff --git a/django_etesync/apps.py b/django_etesync/apps.py deleted file mode 100644 index adb8f96..0000000 --- a/django_etesync/apps.py +++ /dev/null @@ -1,5 +0,0 @@ -from django.apps import AppConfig - - -class DjangoEtesyncConfig(AppConfig): - name = 'django_etesync' From 54268ac0273486e53829d812c6d30dccaac2e214 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 17 Jun 2020 14:08:08 +0300 Subject: [PATCH 193/511] Login: add an action indicator to know the user signed a login request. --- django_etebase/serializers.py | 1 + django_etebase/views.py | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index c940d6e..576431d 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -416,6 +416,7 @@ class AuthenticationLoginSerializer(serializers.Serializer): class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer): challenge = BinaryBase64Field() host = serializers.CharField() + action = serializers.CharField() def create(self, validated_data): raise NotImplementedError() diff --git a/django_etebase/views.py b/django_etebase/views.py index 59fcaa2..2b0ec58 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -607,6 +607,7 @@ class AuthenticationViewSet(viewsets.ViewSet): user = self.get_login_user(username) host = serializer.validated_data['host'] challenge = serializer.validated_data['challenge'] + action = serializer.validated_data['action'] salt = bytes(user.userinfo.salt) enc_key = self.get_encryption_key(salt) @@ -614,7 +615,10 @@ class AuthenticationViewSet(viewsets.ViewSet): challenge_data = json.loads(box.decrypt(challenge).decode()) now = int(datetime.now().timestamp()) - if now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: + if action != "login": + content = {'code': 'wrong_action', 'detail': 'Expected "login" but got something else'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} return Response(content, status=status.HTTP_400_BAD_REQUEST) elif challenge_data['userId'] != user.id: From ab0d85c84fa4ffb5ebd5e797283150a6a6881ab5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 17 Jun 2020 14:38:02 +0300 Subject: [PATCH 194/511] Change password: change to require a signed request, just like login. Without this, it would be sufficient to steal an auth token to render the account unusable because it would be possible to just reset the encrypted content of the account. With this change we require the user to actually know the account password in order to do it. --- django_etebase/serializers.py | 2 +- django_etebase/views.py | 117 +++++++++++++++++++++------------- 2 files changed, 72 insertions(+), 47 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 576431d..c401d91 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -425,7 +425,7 @@ class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer) raise NotImplementedError() -class AuthenticationChangePasswordSerializer(serializers.ModelSerializer): +class AuthenticationChangePasswordInnerSerializer(AuthenticationLoginInnerSerializer): loginPubkey = BinaryBase64Field() encryptedContent = BinaryBase64Field() diff --git a/django_etebase/views.py b/django_etebase/views.py index 2b0ec58..999dc65 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -49,7 +49,7 @@ from .models import ( ) from .serializers import ( b64encode, - AuthenticationChangePasswordSerializer, + AuthenticationChangePasswordInnerSerializer, AuthenticationSignupSerializer, AuthenticationLoginChallengeSerializer, AuthenticationLoginSerializer, @@ -562,6 +562,44 @@ class AuthenticationViewSet(viewsets.ViewSet): kwargs = {User.USERNAME_FIELD: username} return get_object_or_404(self.get_queryset(), **kwargs) + def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): + from datetime import datetime + + username = validated_data.get('username') + user = self.get_login_user(username) + host = validated_data['host'] + challenge = validated_data['challenge'] + action = validated_data['action'] + + salt = bytes(user.userinfo.salt) + enc_key = self.get_encryption_key(salt) + box = nacl.secret.SecretBox(enc_key) + + challenge_data = json.loads(box.decrypt(challenge).decode()) + now = int(datetime.now().timestamp()) + if action != expected_action: + content = {'code': 'wrong_action', 'detail': 'Expected "{}" but got something else'.format(expected_action)} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: + content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif challenge_data['userId'] != user.id: + content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + elif not settings.DEBUG and host != request.get_host(): + detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) + content = {'code': 'wrong_host', 'detail': detail} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + + verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) + + try: + verify_key.verify(response_raw, signature) + except nacl.exceptions.BadSignatureError: + return Response({'code': 'login_bad_signature'}, status=status.HTTP_400_BAD_REQUEST) + + return None + @action_decorator(detail=False, methods=['POST']) def login_challenge(self, request): from datetime import datetime @@ -593,56 +631,29 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST']) def login(self, request): - from datetime import datetime - outer_serializer = AuthenticationLoginSerializer(data=request.data) - if outer_serializer.is_valid(): - response_raw = outer_serializer.validated_data['response'] - response = json.loads(response_raw.decode()) - signature = outer_serializer.validated_data['signature'] - - serializer = AuthenticationLoginInnerSerializer(data=response, context={'host': request.get_host()}) - if serializer.is_valid(): - username = serializer.validated_data.get('username') - user = self.get_login_user(username) - host = serializer.validated_data['host'] - challenge = serializer.validated_data['challenge'] - action = serializer.validated_data['action'] - - salt = bytes(user.userinfo.salt) - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = json.loads(box.decrypt(challenge).decode()) - now = int(datetime.now().timestamp()) - if action != "login": - content = {'code': 'wrong_action', 'detail': 'Expected "login" but got something else'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: - content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif challenge_data['userId'] != user.id: - content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host != request.get_host(): - detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) - content = {'code': 'wrong_host', 'detail': detail} - return Response(content, status=status.HTTP_400_BAD_REQUEST) + outer_serializer.is_valid(raise_exception=True) - verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) + response_raw = outer_serializer.validated_data['response'] + response = json.loads(response_raw.decode()) + signature = outer_serializer.validated_data['signature'] - try: - verify_key.verify(response_raw, signature) - except nacl.exceptions.BadSignatureError: - return Response({'code': 'login_bad_signature'}, status=status.HTTP_400_BAD_REQUEST) + serializer = AuthenticationLoginInnerSerializer(data=response, context={'host': request.get_host()}) + serializer.is_valid(raise_exception=True) - data = self.login_response_data(user) + bad_login_response = self.validate_login_request( + request, serializer.validated_data, response_raw, signature, "login") + if bad_login_response is not None: + return bad_login_response - user_logged_in.send(sender=user.__class__, request=request, user=user) + username = serializer.validated_data.get('username') + user = self.get_login_user(username) - return Response(data, status=status.HTTP_200_OK) + data = self.login_response_data(user) - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + user_logged_in.send(sender=user.__class__, request=request, user=user) + + return Response(data, status=status.HTTP_200_OK) @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) def logout(self, request): @@ -652,11 +663,25 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) def change_password(self, request): - serializer = AuthenticationChangePasswordSerializer(request.user.userinfo, data=request.data) + outer_serializer = AuthenticationLoginSerializer(data=request.data) + outer_serializer.is_valid(raise_exception=True) + + response_raw = outer_serializer.validated_data['response'] + response = json.loads(response_raw.decode()) + signature = outer_serializer.validated_data['signature'] + + serializer = AuthenticationChangePasswordInnerSerializer( + request.user.userinfo, data=response, context={'host': request.get_host()}) serializer.is_valid(raise_exception=True) + + bad_login_response = self.validate_login_request( + request, serializer.validated_data, response_raw, signature, "changePassword") + if bad_login_response is not None: + return bad_login_response + serializer.save() - return Response(status=status.HTTP_200_OK) + return Response({}, status=status.HTTP_200_OK) class TestAuthenticationViewSet(viewsets.ViewSet): From 2d7b90e8486faf6539d3a631890e7b84315c7837 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 18 Jun 2020 16:14:55 +0300 Subject: [PATCH 195/511] Collection items: also show deleted items. This was a mistake. We want deleted items to show because we want to know when things have been deleted when we ask for updates. --- django_etebase/views.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 999dc65..0fb4d5f 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -230,8 +230,7 @@ class CollectionItemViewSet(BaseViewSet): # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = type(self).queryset.filter(collection__pk=collection.pk, uid__isnull=False, - revisions__current=True, - revisions__deleted=False) + revisions__current=True) return queryset From 6117cac1111ae04722463c240b4eb03e7d237add Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 22 Jun 2020 13:03:58 +0300 Subject: [PATCH 196/511] List APIs: return a done field to indicate the fetch is done. --- django_etebase/views.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 0fb4d5f..2944fdb 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -122,8 +122,10 @@ class BaseViewSet(viewsets.ModelViewSet): queryset = queryset[:limit] queryset, new_stoken = self.get_queryset_stoken(queryset) new_stoken = new_stoken or stoken + # This is not the most efficient way of implementing this, but it's good enough + done = len(queryset) < limit - return queryset, new_stoken + return queryset, new_stoken, done # Change how our list works by default def list(self, request, collection_uid=None): @@ -193,13 +195,14 @@ class CollectionViewSet(BaseViewSet): def list(self, request): queryset = self.get_queryset() - queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer(queryset, many=True) ret = { 'data': serializer.data, 'stoken': new_stoken, + 'done': done, } stoken_obj = self.get_stoken_obj(request) @@ -256,13 +259,14 @@ class CollectionItemViewSet(BaseViewSet): def list(self, request, collection_uid=None): queryset = self.get_queryset() - queryset, new_stoken = self.filter_by_stoken_and_limit(request, queryset) + queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer(queryset, many=True) ret = { 'data': serializer.data, 'stoken': new_stoken, + 'done': done, } return Response(ret) @@ -275,6 +279,7 @@ class CollectionItemViewSet(BaseViewSet): serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) ret = { 'data': serializer.data, + 'done': True, # we always return all the items, so it's always done } return Response(ret) @@ -308,6 +313,7 @@ class CollectionItemViewSet(BaseViewSet): ret = { 'data': serializer.data, 'stoken': new_stoken, + 'done': True, # we always return all the items, so it's always done } return Response(ret) From fcb58f0f4ce353c2aa8bef3f68b7d70572ae63fa Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 22 Jun 2020 14:20:26 +0300 Subject: [PATCH 197/511] List APIs: fix the stoken calculation for collections. I'm not sure why it just wouldn't work with aggregate. I also couldn't get it to work with annotate then aggregate or any other alternative. --- django_etebase/views.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 2944fdb..955c49f 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -106,9 +106,15 @@ class BaseViewSet(viewsets.ModelViewSet): return queryset, stoken_rev def get_queryset_stoken(self, queryset): - aggr_fields = {x: Max(x) for x in self.stoken_id_fields} - aggr = queryset.aggregate(**aggr_fields) - maxid = max(map(lambda x: x or -1, aggr.values())) + aggr_field_names = ['max_{}'.format(i) for i, x in enumerate(self.stoken_id_fields)] + aggr_fields = {name: Max(field) for name, field in zip(aggr_field_names, self.stoken_id_fields)} + aggr = queryset.annotate(**aggr_fields).values(*aggr_field_names) + # FIXME: we are doing it in python instead of SQL because I just couldn't get aggregate to work over the + # annotated values. This should probably be fixed as it could be quite slow + maxid = -1 + for row in aggr: + rowmaxid = max(map(lambda x: x or -1, row.values())) + maxid = max(maxid, rowmaxid) new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid).uid return queryset, new_stoken From b4db35bca16d251bca2090876b7b903a4611492f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 22 Jun 2020 17:27:07 +0300 Subject: [PATCH 198/511] List APIs: add done to APIs that didn't have it. --- django_etebase/views.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/django_etebase/views.py b/django_etebase/views.py index 955c49f..0fb6395 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -140,6 +140,7 @@ class BaseViewSet(viewsets.ModelViewSet): ret = { 'data': serializer.data, + 'done': True, # we always return all the items, so it's always done } return Response(ret) @@ -449,6 +450,7 @@ class CollectionMemberViewSet(BaseViewSet): ret = { 'data': serializer.data, + 'done': True, # we always return all the items, so it's always done } return Response(ret) From d5300a76d8fd5c1e6d2154c74689b299a1563211 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 22 Jun 2020 17:51:56 +0300 Subject: [PATCH 199/511] Members: add support for iterators when listing members --- django_etebase/views.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 0fb6395..eb507f2 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -428,6 +428,7 @@ class CollectionMemberViewSet(BaseViewSet): serializer_class = CollectionMemberSerializer lookup_field = 'user__' + User.USERNAME_FIELD lookup_url_kwarg = 'username' + stoken_id_fields = ['stoken__id'] # FIXME: need to make sure that there's always an admin, and maybe also don't let an owner remove adm access # (if we want to transfer, we need to do that specifically) @@ -444,13 +445,24 @@ class CollectionMemberViewSet(BaseViewSet): return queryset.filter(collection=collection) + # We override this method because we expect the stoken to be called iterator + def get_stoken_obj(self, request): + stoken = request.GET.get('iterator', None) + + if stoken is not None: + return get_object_or_404(Stoken.objects.all(), uid=stoken) + + return None + def list(self, request, collection_uid=None): queryset = self.get_queryset() + queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer(queryset, many=True) ret = { 'data': serializer.data, - 'done': True, # we always return all the items, so it's always done + 'iterator': new_stoken, # Here we call it an iterator, it's only stoken for collection/items + 'done': done, } return Response(ret) From 37bae63a466b4ad7bc8eee62817488acabc8327d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 22 Jun 2020 18:26:32 +0300 Subject: [PATCH 200/511] Invitations: add support for fetching using an iterator --- django_etebase/views.py | 40 ++++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index eb507f2..70e635b 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -486,14 +486,42 @@ class CollectionMemberViewSet(BaseViewSet): return Response({}) -class InvitationOutgoingViewSet(BaseViewSet): - allowed_methods = ['GET', 'POST', 'PUT', 'DELETE'] - permission_classes = BaseViewSet.permission_classes +class InvitationBaseViewSet(BaseViewSet): queryset = CollectionInvitation.objects.all() serializer_class = CollectionInvitationSerializer lookup_field = 'uid' lookup_url_kwarg = 'invitation_uid' + def list(self, request, collection_uid=None): + limit = int(request.GET.get('limit', 50)) + iterator = request.GET.get('iterator', None) + + queryset = self.get_queryset().order_by('id') + + if iterator is not None: + iterator = get_object_or_404(queryset, uid=iterator) + queryset = queryset.filter(id__gt=iterator.id) + + queryset = queryset[:limit] + serializer = self.get_serializer(queryset, many=True) + + # This is not the most efficient way of implementing this, but it's good enough + done = len(queryset) < limit + + last_item = len(queryset) > 0 and serializer.data[-1] + + ret = { + 'data': serializer.data, + 'iterator': last_item and last_item['uid'], + 'done': done, + } + + return Response(ret) + + +class InvitationOutgoingViewSet(InvitationBaseViewSet): + allowed_methods = ['GET', 'POST', 'PUT', 'DELETE'] + def get_queryset(self, queryset=None): if queryset is None: queryset = type(self).queryset @@ -528,12 +556,8 @@ class InvitationOutgoingViewSet(BaseViewSet): return Response(serializer.data) -class InvitationIncomingViewSet(BaseViewSet): +class InvitationIncomingViewSet(InvitationBaseViewSet): allowed_methods = ['GET', 'DELETE'] - queryset = CollectionInvitation.objects.all() - serializer_class = CollectionInvitationSerializer - lookup_field = 'uid' - lookup_url_kwarg = 'invitation_uid' def get_queryset(self, queryset=None): if queryset is None: From 267d749c45a52378310705768c3b6685df8744f6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 23 Jun 2020 12:55:28 +0300 Subject: [PATCH 201/511] Collection: change collections to be an extension of items Each collection now has an item and the item's UID is the collections UID. This lets us manipulate collections just like items, and as part of transactions. This is significant because it lets us change them as part of transactions! --- .../migrations/0016_auto_20200623_0820.py | 31 +++++++++++++ django_etebase/models.py | 11 ++--- django_etebase/permissions.py | 6 +-- django_etebase/serializers.py | 43 ++++++++++--------- django_etebase/views.py | 37 ++++++---------- 5 files changed, 72 insertions(+), 56 deletions(-) create mode 100644 django_etebase/migrations/0016_auto_20200623_0820.py diff --git a/django_etebase/migrations/0016_auto_20200623_0820.py b/django_etebase/migrations/0016_auto_20200623_0820.py new file mode 100644 index 0000000..2c11157 --- /dev/null +++ b/django_etebase/migrations/0016_auto_20200623_0820.py @@ -0,0 +1,31 @@ +# Generated by Django 3.0.3 on 2020-06-23 08:20 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0015_collectionitemrevision_salt'), + ] + + operations = [ + migrations.AddField( + model_name='collection', + name='main_item', + field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.PROTECT, related_name='parent', to='django_etebase.CollectionItem'), + ), + migrations.AlterUniqueTogether( + name='collection', + unique_together=set(), + ), + migrations.RemoveField( + model_name='collection', + name='uid', + ), + migrations.RemoveField( + model_name='collection', + name='version', + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 53239e7..2702389 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -27,20 +27,15 @@ UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9]*$', message='Not a valid UID' class Collection(models.Model): - uid = models.CharField(db_index=True, blank=False, null=False, - max_length=43, validators=[UidValidator]) - version = models.PositiveSmallIntegerField() + main_item = models.ForeignKey('CollectionItem', related_name='parent', null=True, on_delete=models.SET_NULL) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) - class Meta: - unique_together = ('uid', 'owner') - def __str__(self): return self.uid @cached_property - def main_item(self): - return self.items.get(uid=None) + def uid(self): + return self.main_item.uid @property def content(self): diff --git a/django_etebase/permissions.py b/django_etebase/permissions.py index a4217a8..6a36afb 100644 --- a/django_etebase/permissions.py +++ b/django_etebase/permissions.py @@ -31,7 +31,7 @@ class IsCollectionAdmin(permissions.BasePermission): def has_permission(self, request, view): collection_uid = view.kwargs['collection_uid'] try: - collection = view.get_collection_queryset().get(uid=collection_uid) + collection = view.get_collection_queryset().get(main_item__uid=collection_uid) return is_collection_admin(collection, request.user) except Collection.DoesNotExist: # If the collection does not exist, we want to 404 later, not permission denied. @@ -53,7 +53,7 @@ class IsCollectionAdminOrReadOnly(permissions.BasePermission): return True try: - collection = view.get_collection_queryset().get(uid=collection_uid) + collection = view.get_collection_queryset().get(main_item__uid=collection_uid) if request.method in permissions.SAFE_METHODS: return True @@ -73,7 +73,7 @@ class HasWriteAccessOrReadOnly(permissions.BasePermission): def has_permission(self, request, view): collection_uid = view.kwargs['collection_uid'] try: - collection = view.get_collection_queryset().get(uid=collection_uid) + collection = view.get_collection_queryset().get(main_item__uid=collection_uid) if request.method in permissions.SAFE_METHODS: return True else: diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index c401d91..c194fdd 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -182,15 +182,19 @@ class CollectionItemBulkGetSerializer(serializers.ModelSerializer): class CollectionSerializer(serializers.ModelSerializer): - encryptionKey = CollectionEncryptionKeyField() + collectionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) + + uid = serializers.CharField(source='main_item.uid') + encryptionKey = BinaryBase64Field(source='main_item.encryptionKey') etag = serializers.CharField(allow_null=True, write_only=True) - content = CollectionItemRevisionSerializer(many=False) + version = serializers.IntegerField(min_value=0, source='main_item.version') + content = CollectionItemRevisionSerializer(many=False, source='main_item.content') class Meta: model = models.Collection - fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'content', 'stoken', 'etag') + fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'collectionKey', 'content', 'stoken', 'etag') def get_access_level_from_context(self, obj): request = self.context.get('request', None) @@ -200,9 +204,16 @@ class CollectionSerializer(serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" + collection_key = validated_data.pop('collectionKey') + etag = validated_data.pop('etag') - revision_data = validated_data.pop('content') - encryption_key = validated_data.pop('encryptionKey') + + main_item_data = validated_data.pop('main_item') + uid = main_item_data.pop('uid') + version = main_item_data.pop('version') + revision_data = main_item_data.pop('content') + encryption_key = main_item_data.pop('encryptionKey') + instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): @@ -211,7 +222,10 @@ class CollectionSerializer(serializers.ModelSerializer): instance.save() main_item = models.CollectionItem.objects.create( - uid=None, encryptionKey=None, version=instance.version, collection=instance) + uid=uid, encryptionKey=encryption_key, version=version, collection=instance) + + instance.main_item = main_item + instance.save() process_revisions_for_item(main_item, revision_data) @@ -219,26 +233,13 @@ class CollectionSerializer(serializers.ModelSerializer): stoken=models.Stoken.objects.create(), user=validated_data.get('owner'), accessLevel=models.AccessLevels.ADMIN, - encryptionKey=encryption_key, + encryptionKey=collection_key, ).save() return instance def update(self, instance, validated_data): - """Function that's called when this serializer is meant to update an item""" - revision_data = validated_data.pop('content') - - with transaction.atomic(): - main_item = instance.main_item - # We don't have to use select_for_update here because the unique constraint on current guards against - # the race condition. But it's a good idea because it'll lock and wait rather than fail. - current_revision = main_item.revisions.filter(current=True).select_for_update().first() - current_revision.current = None - current_revision.save() - - process_revisions_for_item(main_item, revision_data) - - return instance + raise NotImplementedError() class CollectionMemberSerializer(serializers.ModelSerializer): diff --git a/django_etebase/views.py b/django_etebase/views.py index 70e635b..4f5d757 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -147,11 +147,12 @@ class BaseViewSet(viewsets.ModelViewSet): class CollectionViewSet(BaseViewSet): - allowed_methods = ['GET', 'POST', 'DELETE'] + allowed_methods = ['GET', 'POST'] permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly, ) queryset = Collection.objects.all() serializer_class = CollectionSerializer - lookup_field = 'uid' + lookup_field = 'main_item__uid' + lookup_url_kwarg = 'uid' stoken_id_fields = ['items__revisions__stoken__id', 'members__stoken__id'] def get_queryset(self, queryset=None): @@ -173,19 +174,7 @@ class CollectionViewSet(BaseViewSet): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def update(self, request, *args, **kwargs): - instance = self.get_object() - - stoken = request.GET.get('stoken', None) - - if stoken is not None and stoken != instance.stoken: - content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - serializer = self.get_serializer(instance, data=request.data) - serializer.is_valid(raise_exception=True) - self.perform_update(serializer) - - return Response({}) + return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def create(self, request, *args, **kwargs): serializer = self.get_serializer(data=request.data) @@ -216,7 +205,7 @@ class CollectionViewSet(BaseViewSet): if stoken_obj is not None: # FIXME: honour limit? (the limit should be combined for data and this because of stoken) remed = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) \ - .values_list('collection__uid', flat=True) + .values_list('collection__main_item__uid', flat=True) if len(remed) > 0: ret['removedMemberships'] = [{'uid': x} for x in remed] @@ -234,7 +223,7 @@ class CollectionItemViewSet(BaseViewSet): def get_queryset(self): collection_uid = self.kwargs['collection_uid'] try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) except Collection.DoesNotExist: raise Http404("Collection does not exist") # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') @@ -280,7 +269,7 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): # FIXME: need pagination support - col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) col_it = get_object_or_404(col.items, uid=uid) serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) @@ -336,7 +325,7 @@ class CollectionItemViewSet(BaseViewSet): with transaction.atomic(): # We need this for locking on the collection object collection_object = get_object_or_404( self.get_collection_queryset(Collection.objects).select_for_update(), # Lock writes on the collection - uid=collection_uid) + main_item__uid=collection_uid) if stoken is not None and stoken != collection_object.stoken: content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} @@ -388,7 +377,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return queryset.filter(members__user=user) def create(self, request, collection_uid=None, collection_item_uid=None): - col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) serializer = self.get_serializer_class()(data=request.data) @@ -408,7 +397,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): import os from django.views.static import serve - col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) chunk = get_object_or_404(col_it.chunks, uid=uid) @@ -436,7 +425,7 @@ class CollectionMemberViewSet(BaseViewSet): def get_queryset(self, queryset=None): collection_uid = self.kwargs['collection_uid'] try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) except Collection.DoesNotExist: raise Http404('Collection does not exist') @@ -478,7 +467,7 @@ class CollectionMemberViewSet(BaseViewSet): @action_decorator(detail=False, methods=['POST'], permission_classes=our_base_permission_classes) def leave(self, request, collection_uid=None): collection_uid = self.kwargs['collection_uid'] - col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) member = col.members.get(user=request.user) self.perform_destroy(member) @@ -534,7 +523,7 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): collection_uid = serializer.validated_data.get('collection', {}).get('uid') try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) + collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) except Collection.DoesNotExist: raise Http404('Collection does not exist') From 291ebaa3f77d99d2a1ae8ef19d5950b52810117b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 23 Jun 2020 13:00:51 +0300 Subject: [PATCH 202/511] Items must have a uid now (not null). This is due to the previous change. --- .../migrations/0017_auto_20200623_0958.py | 25 +++++++++++++++++++ django_etebase/models.py | 2 +- django_etebase/views.py | 1 - 3 files changed, 26 insertions(+), 2 deletions(-) create mode 100644 django_etebase/migrations/0017_auto_20200623_0958.py diff --git a/django_etebase/migrations/0017_auto_20200623_0958.py b/django_etebase/migrations/0017_auto_20200623_0958.py new file mode 100644 index 0000000..e244b13 --- /dev/null +++ b/django_etebase/migrations/0017_auto_20200623_0958.py @@ -0,0 +1,25 @@ +# Generated by Django 3.0.3 on 2020-06-23 09:58 + +import django.core.validators +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0016_auto_20200623_0820'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='main_item', + field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='parent', to='django_etebase.CollectionItem'), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9]*$')]), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 2702389..87b0ed1 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -58,7 +58,7 @@ class Collection(models.Model): class CollectionItem(models.Model): - uid = models.CharField(db_index=True, blank=False, null=True, + uid = models.CharField(db_index=True, blank=False, max_length=43, validators=[UidValidator]) collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) version = models.PositiveSmallIntegerField() diff --git a/django_etebase/views.py b/django_etebase/views.py index 4f5d757..2e4bf3c 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -228,7 +228,6 @@ class CollectionItemViewSet(BaseViewSet): raise Http404("Collection does not exist") # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = type(self).queryset.filter(collection__pk=collection.pk, - uid__isnull=False, revisions__current=True) return queryset From 317c492688dab64bd2537ac231d3c9621af5aa1f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 23 Jun 2020 13:02:45 +0300 Subject: [PATCH 203/511] CollectionItem: add support for filtering collections' main items. This used to be the default, so it still is. It only affects the list endpoint, the rest all support withCollection anyway, because IDs are passed directly. --- django_etebase/views.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/django_etebase/views.py b/django_etebase/views.py index 2e4bf3c..577d923 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -254,6 +254,10 @@ class CollectionItemViewSet(BaseViewSet): def list(self, request, collection_uid=None): queryset = self.get_queryset() + + if not self.request.query_params.get('withCollection', False): + queryset = queryset.filter(parent__isnull=True) + queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer(queryset, many=True) From 786948c4568e615f5c6465fb9bac264fc68fe711 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 23 Jun 2020 18:04:49 +0300 Subject: [PATCH 204/511] Item revisions: never return the current revision, only old ones. --- django_etebase/views.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 577d923..f44ff5c 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -275,7 +275,8 @@ class CollectionItemViewSet(BaseViewSet): col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) col_it = get_object_or_404(col.items, uid=uid) - serializer = CollectionItemRevisionSerializer(col_it.revisions.order_by('-id'), many=True) + revisions = col_it.revisions.exclude(current=True).order_by('-id') + serializer = CollectionItemRevisionSerializer(revisions, many=True) ret = { 'data': serializer.data, 'done': True, # we always return all the items, so it's always done From 7183b975419096401910d53ba622888e0879bdf2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 23 Jun 2020 18:25:23 +0300 Subject: [PATCH 205/511] Collection revision: implement iteration. --- django_etebase/views.py | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index f44ff5c..0e153a4 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -271,15 +271,30 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=True, methods=['GET']) def revision(self, request, collection_uid=None, uid=None): - # FIXME: need pagination support col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) - col_it = get_object_or_404(col.items, uid=uid) + item = get_object_or_404(col.items, uid=uid) + + limit = int(request.GET.get('limit', 50)) + iterator = request.GET.get('iterator', None) + + queryset = item.revisions.exclude(current=True).order_by('-id') + + if iterator is not None: + iterator = get_object_or_404(queryset, uid=iterator) + queryset = queryset.filter(id__lt=iterator.id) + + queryset = queryset[:limit] + serializer = CollectionItemRevisionSerializer(queryset, many=True) + + # This is not the most efficient way of implementing this, but it's good enough + done = len(queryset) < limit + + last_item = len(queryset) > 0 and serializer.data[-1] - revisions = col_it.revisions.exclude(current=True).order_by('-id') - serializer = CollectionItemRevisionSerializer(revisions, many=True) ret = { 'data': serializer.data, - 'done': True, # we always return all the items, so it's always done + 'iterator': last_item and last_item['uid'], + 'done': done, } return Response(ret) From 68365f5d75c7c2cae14dad93c603430f4b2703eb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 23 Jun 2020 18:35:09 +0300 Subject: [PATCH 206/511] Collection revision: support the inline parameter. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 0e153a4..f692ce0 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -284,7 +284,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = queryset.filter(id__lt=iterator.id) queryset = queryset[:limit] - serializer = CollectionItemRevisionSerializer(queryset, many=True) + serializer = CollectionItemRevisionSerializer(queryset, context=self.get_serializer_context(), many=True) # This is not the most efficient way of implementing this, but it's good enough done = len(queryset) < limit From 2da49bb95e8b7b29fdfec04807a895d22b6f99ff Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 10:02:55 +0300 Subject: [PATCH 207/511] Item revisions: don't exclude current, let the client decide. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index f692ce0..bbd4e3e 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -277,7 +277,7 @@ class CollectionItemViewSet(BaseViewSet): limit = int(request.GET.get('limit', 50)) iterator = request.GET.get('iterator', None) - queryset = item.revisions.exclude(current=True).order_by('-id') + queryset = item.revisions.order_by('-id') if iterator is not None: iterator = get_object_or_404(queryset, uid=iterator) From 1bed39af9d3d4205d15d42947cbedc6a52cb1fc2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 10:48:47 +0300 Subject: [PATCH 208/511] Collection/item uid: allow base64-url not just base62. --- .../migrations/0018_auto_20200624_0748.py | 19 +++++++++++++++++++ django_etebase/models.py | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 django_etebase/migrations/0018_auto_20200624_0748.py diff --git a/django_etebase/migrations/0018_auto_20200624_0748.py b/django_etebase/migrations/0018_auto_20200624_0748.py new file mode 100644 index 0000000..ec59e0c --- /dev/null +++ b/django_etebase/migrations/0018_auto_20200624_0748.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-06-24 07:48 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0017_auto_20200623_0958'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]*$')]), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 87b0ed1..5f7a6f2 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -23,7 +23,7 @@ from django.utils.crypto import get_random_string Base64Url256BitlValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{42,43}$', message='Expected a base64url.') -UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9]*$', message='Not a valid UID') +UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]*$', message='Not a valid UID') class Collection(models.Model): From f6ef514661a2aef45a92f368a1ad7296fe65814a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 10:58:27 +0300 Subject: [PATCH 209/511] Collection members: order by id so order is consistent. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index bbd4e3e..5f1cbff 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -463,7 +463,7 @@ class CollectionMemberViewSet(BaseViewSet): return None def list(self, request, collection_uid=None): - queryset = self.get_queryset() + queryset = self.get_queryset().order_by('id') queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer(queryset, many=True) From 0a19cd7e2c84004a2f575f587737f0d90312e915 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 11:30:37 +0300 Subject: [PATCH 210/511] Stoken filtering: abstract getting the stoken id. --- django_etebase/views.py | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 5f1cbff..5aa4b26 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -88,8 +88,11 @@ class BaseViewSet(viewsets.ModelViewSet): user = self.request.user return queryset.filter(members__user=user) + def get_stoken_obj_id(self, request): + return request.GET.get('stoken', None) + def get_stoken_obj(self, request): - stoken = request.GET.get('stoken', None) + stoken = self.get_stoken_obj_id(request) if stoken is not None: return get_object_or_404(Stoken.objects.all(), uid=stoken) @@ -454,13 +457,8 @@ class CollectionMemberViewSet(BaseViewSet): return queryset.filter(collection=collection) # We override this method because we expect the stoken to be called iterator - def get_stoken_obj(self, request): - stoken = request.GET.get('iterator', None) - - if stoken is not None: - return get_object_or_404(Stoken.objects.all(), uid=stoken) - - return None + def get_stoken_obj_id(self, request): + return request.GET.get('iterator', None) def list(self, request, collection_uid=None): queryset = self.get_queryset().order_by('id') From caa84c2a96a1e2d456b65893292744e4feab24eb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 13:20:07 +0300 Subject: [PATCH 211/511] Stoken filtering: clean up stoken filtering and annotation. We are now querying the database less and simplified the queries. --- django_etebase/views.py | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 5aa4b26..4c528f1 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -19,7 +19,8 @@ from django.conf import settings from django.contrib.auth import get_user_model, user_logged_in, user_logged_out from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError -from django.db.models import Max, Q +from django.db.models import Max, Q, F, Value as V +from django.db.models.functions import Coalesce, Greatest from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -101,22 +102,20 @@ class BaseViewSet(viewsets.ModelViewSet): def filter_by_stoken(self, request, queryset): stoken_rev = self.get_stoken_obj(request) + + aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] + max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] + queryset = queryset.annotate(max_stoken=max_stoken) + if stoken_rev is not None: - filter_by_map = map(lambda x: Q(**{x + '__gt': stoken_rev.id}), self.stoken_id_fields) - filter_by = reduce(lambda x, y: x | y, filter_by_map) - queryset = queryset.filter(filter_by).distinct() + queryset = queryset.filter(max_stoken__gt=stoken_rev.id) return queryset, stoken_rev def get_queryset_stoken(self, queryset): - aggr_field_names = ['max_{}'.format(i) for i, x in enumerate(self.stoken_id_fields)] - aggr_fields = {name: Max(field) for name, field in zip(aggr_field_names, self.stoken_id_fields)} - aggr = queryset.annotate(**aggr_fields).values(*aggr_field_names) - # FIXME: we are doing it in python instead of SQL because I just couldn't get aggregate to work over the - # annotated values. This should probably be fixed as it could be quite slow maxid = -1 - for row in aggr: - rowmaxid = max(map(lambda x: x or -1, row.values())) + for row in queryset: + rowmaxid = getattr(row, 'max_stoken') or -1 maxid = max(maxid, rowmaxid) new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid).uid From 61383b98965b788aa9a1fde3c18e1f2c1f02c537 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 13:35:23 +0300 Subject: [PATCH 212/511] Stoken filtering: order by max_stoken to make sure we have a reliable order. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 4c528f1..f1e3802 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -105,7 +105,7 @@ class BaseViewSet(viewsets.ModelViewSet): aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] - queryset = queryset.annotate(max_stoken=max_stoken) + queryset = queryset.annotate(max_stoken=max_stoken).order_by('max_stoken') if stoken_rev is not None: queryset = queryset.filter(max_stoken__gt=stoken_rev.id) From 0ce2e8d99685ba500dd34fb55d18ea293052e618 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 14:34:03 +0300 Subject: [PATCH 213/511] Filter by stoken: cleanup and fix the done implementation The done implementation wasn't great because it would indicate we are not done even when we are when the last chunk returned is exactly the size of limit. --- django_etebase/views.py | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index f1e3802..68353b4 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -119,7 +119,7 @@ class BaseViewSet(viewsets.ModelViewSet): maxid = max(maxid, rowmaxid) new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid).uid - return queryset, new_stoken + return new_stoken def filter_by_stoken_and_limit(self, request, queryset): limit = int(request.GET.get('limit', 50)) @@ -127,13 +127,16 @@ class BaseViewSet(viewsets.ModelViewSet): queryset, stoken_rev = self.filter_by_stoken(request, queryset) stoken = stoken_rev.uid if stoken_rev is not None else None - queryset = queryset[:limit] - queryset, new_stoken = self.get_queryset_stoken(queryset) - new_stoken = new_stoken or stoken - # This is not the most efficient way of implementing this, but it's good enough - done = len(queryset) < limit + result = list(queryset[:limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + new_stoken = self.get_queryset_stoken(result) or stoken - return queryset, new_stoken, done + return result, new_stoken, done # Change how our list works by default def list(self, request, collection_uid=None): @@ -193,9 +196,9 @@ class CollectionViewSet(BaseViewSet): def list(self, request): queryset = self.get_queryset() - queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) + result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) - serializer = self.get_serializer(queryset, many=True) + serializer = self.get_serializer(result, many=True) ret = { 'data': serializer.data, @@ -260,9 +263,9 @@ class CollectionItemViewSet(BaseViewSet): if not self.request.query_params.get('withCollection', False): queryset = queryset.filter(parent__isnull=True) - queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) + result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) - serializer = self.get_serializer(queryset, many=True) + serializer = self.get_serializer(result, many=True) ret = { 'data': serializer.data, @@ -321,7 +324,7 @@ class CollectionItemViewSet(BaseViewSet): revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) - queryset, new_stoken = self.get_queryset_stoken(queryset) + new_stoken = self.get_queryset_stoken(queryset) stoken = stoken_rev and stoken_rev.uid new_stoken = new_stoken or stoken @@ -461,8 +464,8 @@ class CollectionMemberViewSet(BaseViewSet): def list(self, request, collection_uid=None): queryset = self.get_queryset().order_by('id') - queryset, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) - serializer = self.get_serializer(queryset, many=True) + result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) + serializer = self.get_serializer(result, many=True) ret = { 'data': serializer.data, From c21c6af1d760be18b81bbe030b084462f13ce8df Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 14:38:29 +0300 Subject: [PATCH 214/511] Filter by stoken: fix the done implementation for more functions The done implementation wasn't great because it would indicate we are not done even when we are when the last chunk returned is exactly the size of limit. --- django_etebase/views.py | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 68353b4..0606e5b 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -288,13 +288,16 @@ class CollectionItemViewSet(BaseViewSet): iterator = get_object_or_404(queryset, uid=iterator) queryset = queryset.filter(id__lt=iterator.id) - queryset = queryset[:limit] - serializer = CollectionItemRevisionSerializer(queryset, context=self.get_serializer_context(), many=True) + result = list(queryset[:limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] - # This is not the most efficient way of implementing this, but it's good enough - done = len(queryset) < limit + serializer = CollectionItemRevisionSerializer(result, context=self.get_serializer_context(), many=True) - last_item = len(queryset) > 0 and serializer.data[-1] + last_item = len(result) > 0 and serializer.data[-1] ret = { 'data': serializer.data, @@ -510,13 +513,16 @@ class InvitationBaseViewSet(BaseViewSet): iterator = get_object_or_404(queryset, uid=iterator) queryset = queryset.filter(id__gt=iterator.id) - queryset = queryset[:limit] - serializer = self.get_serializer(queryset, many=True) + result = list(queryset[:limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] - # This is not the most efficient way of implementing this, but it's good enough - done = len(queryset) < limit + serializer = self.get_serializer(result, many=True) - last_item = len(queryset) > 0 and serializer.data[-1] + last_item = len(result) > 0 and serializer.data[-1] ret = { 'data': serializer.data, From cbb1d81850a514250650b1fcc4f0985c3b1adf71 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 24 Jun 2020 15:55:36 +0300 Subject: [PATCH 215/511] Rename inline to prefetch and have it on by default. --- django_etebase/serializers.py | 3 +-- django_etebase/views.py | 8 ++++---- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index c194fdd..9ca33b1 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -83,8 +83,7 @@ class CollectionContentField(BinaryBase64Field): class ChunksField(serializers.RelatedField): def to_representation(self, obj): obj = obj.chunk - inline = self.context.get('inline', False) - if inline: + if self.context.get('prefetch'): with open(obj.chunkFile.path, 'rb') as f: return (obj.uid, b64encode(f.read())) else: diff --git a/django_etebase/views.py b/django_etebase/views.py index 0606e5b..64acb18 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -167,8 +167,8 @@ class CollectionViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - inline = 'inline' in self.request.query_params - context.update({'request': self.request, 'inline': inline}) + prefetch = self.request.query_params.get('prefetch', True) + context.update({'request': self.request, 'prefetch': prefetch}) return context def destroy(self, request, uid=None): @@ -239,8 +239,8 @@ class CollectionItemViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - inline = 'inline' in self.request.query_params - context.update({'request': self.request, 'inline': inline}) + prefetch = self.request.query_params.get('prefetch', True) + context.update({'request': self.request, 'prefetch': prefetch}) return context def create(self, request, collection_uid=None): From 625df229895d6542485c93828cc87a299490541e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 26 Jun 2020 10:31:03 +0300 Subject: [PATCH 216/511] Make item encryption key optional for collections/items Collections still have a unique encryption key (their collection key), and items just have a unique key per item in a collection that's derived from the main key and if we ever want to share items across collections or do something fancy like that we can just add an encrypted key in there. --- django_etebase/serializers.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 9ca33b1..a76f388 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -116,7 +116,7 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): class CollectionItemSerializer(serializers.ModelSerializer): - encryptionKey = BinaryBase64Field() + encryptionKey = BinaryBase64Field(required=False, default=None) etag = serializers.CharField(allow_null=True, write_only=True) content = CollectionItemRevisionSerializer(many=False) @@ -186,7 +186,7 @@ class CollectionSerializer(serializers.ModelSerializer): stoken = serializers.CharField(read_only=True) uid = serializers.CharField(source='main_item.uid') - encryptionKey = BinaryBase64Field(source='main_item.encryptionKey') + encryptionKey = BinaryBase64Field(source='main_item.encryptionKey', required=False, default=None) etag = serializers.CharField(allow_null=True, write_only=True) version = serializers.IntegerField(min_value=0, source='main_item.version') content = CollectionItemRevisionSerializer(many=False, source='main_item.content') From 2b52eec41f5278428aa78735042258b6503bd408 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 26 Jun 2020 11:05:01 +0300 Subject: [PATCH 217/511] Allow chunk UIDs to be longer. --- .../migrations/0019_auto_20200626_0748.py | 19 +++++++++++++++++++ django_etebase/models.py | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 django_etebase/migrations/0019_auto_20200626_0748.py diff --git a/django_etebase/migrations/0019_auto_20200626_0748.py b/django_etebase/migrations/0019_auto_20200626_0748.py new file mode 100644 index 0000000..991ca50 --- /dev/null +++ b/django_etebase/migrations/0019_auto_20200626_0748.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-06-26 07:48 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0018_auto_20200624_0748'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=60, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]*$')]), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 5f7a6f2..964012a 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -89,7 +89,7 @@ def chunk_directory_path(instance, filename): class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, - max_length=43, validators=[Base64Url256BitlValidator]) + max_length=60, validators=[UidValidator]) item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) From c00cf501632692ada572f4a2e991717f81d78a74 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 26 Jun 2020 11:21:53 +0300 Subject: [PATCH 218/511] Revision: remove salt field. It's not really needed. More information in the respective change in the js client. --- .../0020_remove_collectionitemrevision_salt.py | 17 +++++++++++++++++ django_etebase/models.py | 1 - django_etebase/serializers.py | 3 +-- 3 files changed, 18 insertions(+), 3 deletions(-) create mode 100644 django_etebase/migrations/0020_remove_collectionitemrevision_salt.py diff --git a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py new file mode 100644 index 0000000..2df32bf --- /dev/null +++ b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py @@ -0,0 +1,17 @@ +# Generated by Django 3.0.3 on 2020-06-26 08:19 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0019_auto_20200626_0748'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectionitemrevision', + name='salt', + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 964012a..c8ceaba 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -110,7 +110,6 @@ class CollectionItemRevision(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) uid = models.CharField(db_index=True, unique=True, blank=False, null=False, max_length=43, validators=[Base64Url256BitlValidator]) - salt = models.BinaryField(editable=True, blank=False, null=False, default=b'') item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index a76f388..2fe0a21 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -107,12 +107,11 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): queryset=models.RevisionChunkRelation.objects.all(), many=True ) - salt = BinaryBase64Field() meta = BinaryBase64Field() class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'meta', 'uid', 'salt', 'deleted') + fields = ('chunks', 'meta', 'uid', 'deleted') class CollectionItemSerializer(serializers.ModelSerializer): From 785e4fae979b5462f73e7abe23644cf5103639e0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 26 Jun 2020 12:13:50 +0300 Subject: [PATCH 219/511] Merge the uidvalidator with the base64url validator and set a min length. --- .../migrations/0021_auto_20200626_0913.py | 40 +++++++++++++++++++ django_etebase/models.py | 9 ++--- 2 files changed, 44 insertions(+), 5 deletions(-) create mode 100644 django_etebase/migrations/0021_auto_20200626_0913.py diff --git a/django_etebase/migrations/0021_auto_20200626_0913.py b/django_etebase/migrations/0021_auto_20200626_0913.py new file mode 100644 index 0000000..b890384 --- /dev/null +++ b/django_etebase/migrations/0021_auto_20200626_0913.py @@ -0,0 +1,40 @@ +# Generated by Django 3.0.3 on 2020-06-26 09:13 + +import django.core.validators +from django.db import migrations, models +import django_etebase.models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0020_remove_collectionitemrevision_salt'), + ] + + operations = [ + migrations.AlterField( + model_name='collectioninvitation', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + migrations.AlterField( + model_name='collectionitem', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + migrations.AlterField( + model_name='collectionitemchunk', + name='uid', + field=models.CharField(db_index=True, max_length=60, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + migrations.AlterField( + model_name='collectionitemrevision', + name='uid', + field=models.CharField(db_index=True, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + migrations.AlterField( + model_name='stoken', + name='uid', + field=models.CharField(db_index=True, default=django_etebase.models.generate_stoken_uid, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index c8ceaba..b4b04fc 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -22,8 +22,7 @@ from django.utils.functional import cached_property from django.utils.crypto import get_random_string -Base64Url256BitlValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{42,43}$', message='Expected a base64url.') -UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]*$', message='Not a valid UID') +UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{20,}$', message='Not a valid UID') class Collection(models.Model): @@ -103,13 +102,13 @@ def generate_stoken_uid(): class Stoken(models.Model): uid = models.CharField(db_index=True, unique=True, blank=False, null=False, default=generate_stoken_uid, - max_length=43, validators=[Base64Url256BitlValidator]) + max_length=43, validators=[UidValidator]) class CollectionItemRevision(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) uid = models.CharField(db_index=True, unique=True, blank=False, null=False, - max_length=43, validators=[Base64Url256BitlValidator]) + max_length=43, validators=[UidValidator]) item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) @@ -179,7 +178,7 @@ class CollectionMemberRemoved(models.Model): class CollectionInvitation(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, - max_length=43, validators=[Base64Url256BitlValidator]) + max_length=43, validators=[UidValidator]) version = models.PositiveSmallIntegerField(default=1) fromMember = models.ForeignKey(CollectionMember, on_delete=models.CASCADE) # FIXME: make sure to delete all invitations for the same collection once one is accepted From 4948e91c65db0842d9c4d50929549df9353a2b65 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 28 Jun 2020 16:52:14 +0300 Subject: [PATCH 220/511] django_etebase: make migration generic and not depend on myauth. --- django_etebase/migrations/0002_userinfo.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/migrations/0002_userinfo.py b/django_etebase/migrations/0002_userinfo.py index 6da0bb8..bfeb2cf 100644 --- a/django_etebase/migrations/0002_userinfo.py +++ b/django_etebase/migrations/0002_userinfo.py @@ -8,7 +8,7 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('myauth', '0001_initial'), + # XXX removed this to make this migration generic ('myauth', '0001_initial'), ('django_etebase', '0001_initial'), ] From 85de674ee28695474237b214de570682f0993c44 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 28 Jun 2020 17:11:20 +0300 Subject: [PATCH 221/511] Move the etebase urls configuration to django_etebase. --- django_etebase/urls.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 django_etebase/urls.py diff --git a/django_etebase/urls.py b/django_etebase/urls.py new file mode 100644 index 0000000..f6d982e --- /dev/null +++ b/django_etebase/urls.py @@ -0,0 +1,30 @@ +from django.conf import settings +from django.conf.urls import include +from django.urls import path + +from rest_framework_nested import routers + +from django_etebase import views + +router = routers.DefaultRouter() +router.register(r'collection', views.CollectionViewSet) +router.register(r'authentication', views.AuthenticationViewSet, basename='authentication') +router.register(r'invitation/incoming', views.InvitationIncomingViewSet, basename='invitation_incoming') +router.register(r'invitation/outgoing', views.InvitationOutgoingViewSet, basename='invitation_outgoing') + +collections_router = routers.NestedSimpleRouter(router, r'collection', lookup='collection') +collections_router.register(r'item', views.CollectionItemViewSet, basename='collection_item') +collections_router.register(r'member', views.CollectionMemberViewSet, basename='collection_member') + +item_router = routers.NestedSimpleRouter(collections_router, r'item', lookup='collection_item') +item_router.register(r'chunk', views.CollectionItemChunkViewSet, basename='collection_items_chunk') + +if settings.DEBUG: + router.register(r'test/authentication', views.TestAuthenticationViewSet, basename='test_authentication') + +app_name = 'django_etebase' +urlpatterns = [ + path('v1/', include(router.urls)), + path('v1/', include(collections_router.urls)), + path('v1/', include(item_router.urls)), +] From 453275eadf1e97b56d54bf3499693d3992af18f8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 29 Jun 2020 11:30:59 +0300 Subject: [PATCH 222/511] Authentication: move to msgpack for the encrypted parts. --- django_etebase/serializers.py | 11 ++++++++++- django_etebase/views.py | 26 +++++++++++++++++--------- requirements.in/base.txt | 1 + requirements.txt | 1 + 4 files changed, 29 insertions(+), 10 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 2fe0a21..e7c0d50 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -64,6 +64,15 @@ class BinaryBase64Field(serializers.Field): return b64decode(data) +# This field does nothing to the data. It's useful for raw binary data +class RawField(serializers.Field): + def to_representation(self, value): + return value + + def to_internal_value(self, data): + return data + + class CollectionEncryptionKeyField(BinaryBase64Field): def get_attribute(self, instance): request = self.context.get('request', None) @@ -413,7 +422,7 @@ class AuthenticationLoginSerializer(serializers.Serializer): class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer): - challenge = BinaryBase64Field() + challenge = RawField() host = serializers.CharField() action = serializers.CharField() diff --git a/django_etebase/views.py b/django_etebase/views.py index 64acb18..c7fdab5 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -12,7 +12,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -import json +import msgpack from functools import reduce from django.conf import settings @@ -72,6 +72,14 @@ from .serializers import ( User = get_user_model() +def msgpack_encode(content): + return msgpack.packb(content, use_bin_type=True) + + +def msgpack_decode(content): + return msgpack.unpackb(content, raw=False) + + class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) @@ -638,7 +646,7 @@ class AuthenticationViewSet(viewsets.ViewSet): enc_key = self.get_encryption_key(salt) box = nacl.secret.SecretBox(enc_key) - challenge_data = json.loads(box.decrypt(challenge).decode()) + challenge_data = msgpack_decode(box.decrypt(challenge)) now = int(datetime.now().timestamp()) if action != expected_action: content = {'code': 'wrong_action', 'detail': 'Expected "{}" but got something else'.format(expected_action)} @@ -680,8 +688,7 @@ class AuthenticationViewSet(viewsets.ViewSet): "timestamp": int(datetime.now().timestamp()), "userId": user.id, } - challenge = box.encrypt(json.dumps( - challenge_data, separators=(',', ':')).encode(), encoder=nacl.encoding.RawEncoder) + challenge = box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder) ret = { "salt": b64encode(salt), @@ -698,10 +705,11 @@ class AuthenticationViewSet(viewsets.ViewSet): outer_serializer.is_valid(raise_exception=True) response_raw = outer_serializer.validated_data['response'] - response = json.loads(response_raw.decode()) + response = msgpack_decode(response_raw) signature = outer_serializer.validated_data['signature'] - serializer = AuthenticationLoginInnerSerializer(data=response, context={'host': request.get_host()}) + context = {'host': request.get_host(), 'supports_binary': True} + serializer = AuthenticationLoginInnerSerializer(data=response, context=context) serializer.is_valid(raise_exception=True) bad_login_response = self.validate_login_request( @@ -730,11 +738,11 @@ class AuthenticationViewSet(viewsets.ViewSet): outer_serializer.is_valid(raise_exception=True) response_raw = outer_serializer.validated_data['response'] - response = json.loads(response_raw.decode()) + response = msgpack_decode(response_raw) signature = outer_serializer.validated_data['signature'] - serializer = AuthenticationChangePasswordInnerSerializer( - request.user.userinfo, data=response, context={'host': request.get_host()}) + context = {'host': request.get_host(), 'supports_binary': True} + serializer = AuthenticationChangePasswordInnerSerializer(request.user.userinfo, data=response, context=context) serializer.is_valid(raise_exception=True) bad_login_response = self.validate_login_request( diff --git a/requirements.in/base.txt b/requirements.in/base.txt index 1ab0e9a..d27e110 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -4,5 +4,6 @@ django-cors-headers django-fullurl djangorestframework drf-nested-routers +msgpack psycopg2-binary pynacl diff --git a/requirements.txt b/requirements.txt index 51b4d35..cd61ff1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -15,6 +15,7 @@ django==3.0.3 # via -r requirements.in/base.txt, django-anymail, dja djangorestframework==3.11.0 # via -r requirements.in/base.txt, drf-nested-routers drf-nested-routers==0.91 # via -r requirements.in/base.txt idna==2.8 # via requests +msgpack==1.0.0 # via -r requirements.in/base.txt psycopg2-binary==2.8.4 # via -r requirements.in/base.txt pycparser==2.20 # via cffi pynacl==1.3.0 # via -r requirements.in/base.txt From fbf5552a62b770bdf5b2246da128b065ae75c9df Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 29 Jun 2020 13:20:23 +0300 Subject: [PATCH 223/511] Modify binary64 field to support binary renderers/parsers Fixes 39c1dfc53c30e65bcbff9e0ba0bb07bfc8bfc577 --- django_etebase/serializers.py | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index e7c0d50..1876572 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -58,19 +58,16 @@ def b64decode(data): class BinaryBase64Field(serializers.Field): def to_representation(self, value): - return b64encode(value) - - def to_internal_value(self, data): - return b64decode(data) - - -# This field does nothing to the data. It's useful for raw binary data -class RawField(serializers.Field): - def to_representation(self, value): - return value + if self.context.get('supports_binary', False): + return value + else: + return b64encode(value) def to_internal_value(self, data): - return data + if isinstance(data, bytes): + return data + else: + return b64decode(data) class CollectionEncryptionKeyField(BinaryBase64Field): @@ -422,7 +419,7 @@ class AuthenticationLoginSerializer(serializers.Serializer): class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer): - challenge = RawField() + challenge = BinaryBase64Field() host = serializers.CharField() action = serializers.CharField() From 2880673e27e41126416d8c642fb6851880eaa8e2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 29 Jun 2020 13:01:40 +0300 Subject: [PATCH 224/511] drf_msgpack: add code to parse/serialise msgpack It's not actually used by clients but it's there and can be used. It works for receiving msgpack messages, but doesn't yet work for sending because some of the types will be converted to base64. --- django_etebase/drf_msgpack/__init__.py | 0 django_etebase/drf_msgpack/apps.py | 5 +++++ django_etebase/drf_msgpack/migrations/__init__.py | 0 django_etebase/drf_msgpack/parsers.py | 14 ++++++++++++++ django_etebase/drf_msgpack/renderers.py | 15 +++++++++++++++ django_etebase/drf_msgpack/views.py | 3 +++ django_etebase/views.py | 15 +++++++++++++-- 7 files changed, 50 insertions(+), 2 deletions(-) create mode 100644 django_etebase/drf_msgpack/__init__.py create mode 100644 django_etebase/drf_msgpack/apps.py create mode 100644 django_etebase/drf_msgpack/migrations/__init__.py create mode 100644 django_etebase/drf_msgpack/parsers.py create mode 100644 django_etebase/drf_msgpack/renderers.py create mode 100644 django_etebase/drf_msgpack/views.py diff --git a/django_etebase/drf_msgpack/__init__.py b/django_etebase/drf_msgpack/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etebase/drf_msgpack/apps.py b/django_etebase/drf_msgpack/apps.py new file mode 100644 index 0000000..619e3e0 --- /dev/null +++ b/django_etebase/drf_msgpack/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class DrfMsgpackConfig(AppConfig): + name = 'drf_msgpack' diff --git a/django_etebase/drf_msgpack/migrations/__init__.py b/django_etebase/drf_msgpack/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etebase/drf_msgpack/parsers.py b/django_etebase/drf_msgpack/parsers.py new file mode 100644 index 0000000..44cd33b --- /dev/null +++ b/django_etebase/drf_msgpack/parsers.py @@ -0,0 +1,14 @@ +import msgpack + +from rest_framework.parsers import BaseParser +from rest_framework.exceptions import ParseError + + +class MessagePackParser(BaseParser): + media_type = 'application/msgpack' + + def parse(self, stream, media_type=None, parser_context=None): + try: + return msgpack.unpackb(stream.read(), raw=False) + except Exception as exc: + raise ParseError('MessagePack parse error - %s' % str(exc)) diff --git a/django_etebase/drf_msgpack/renderers.py b/django_etebase/drf_msgpack/renderers.py new file mode 100644 index 0000000..9445231 --- /dev/null +++ b/django_etebase/drf_msgpack/renderers.py @@ -0,0 +1,15 @@ +import msgpack + +from rest_framework.renderers import BaseRenderer + + +class MessagePackRenderer(BaseRenderer): + media_type = 'application/msgpack' + format = 'msgpack' + render_style = 'binary' + charset = None + + def render(self, data, media_type=None, renderer_context=None): + if data is None: + return b'' + return msgpack.packb(data, use_bin_type=True) diff --git a/django_etebase/drf_msgpack/views.py b/django_etebase/drf_msgpack/views.py new file mode 100644 index 0000000..91ea44a --- /dev/null +++ b/django_etebase/drf_msgpack/views.py @@ -0,0 +1,3 @@ +from django.shortcuts import render + +# Create your views here. diff --git a/django_etebase/views.py b/django_etebase/views.py index c7fdab5..93a7645 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -26,9 +26,10 @@ from django.shortcuts import get_object_or_404 from rest_framework import status from rest_framework import viewsets -from rest_framework import parsers from rest_framework.decorators import action as action_decorator from rest_framework.response import Response +from rest_framework.parsers import JSONParser, FormParser, MultiPartParser +from rest_framework.renderers import JSONRenderer, BrowsableAPIRenderer import nacl.encoding import nacl.signing @@ -37,6 +38,9 @@ import nacl.hash from .token_auth.models import AuthToken +from .drf_msgpack.parsers import MessagePackParser +from .drf_msgpack.renderers import MessagePackRenderer + from . import app_settings, permissions from .models import ( Collection, @@ -83,6 +87,8 @@ def msgpack_decode(content): class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) + renderer_classes = [JSONRenderer, MessagePackRenderer, BrowsableAPIRenderer] + parser_classes = [JSONParser, MessagePackParser, FormParser, MultiPartParser] stoken_id_fields = None def get_serializer_class(self): @@ -398,9 +404,10 @@ class CollectionItemViewSet(BaseViewSet): class CollectionItemChunkViewSet(viewsets.ViewSet): allowed_methods = ['GET', 'POST'] - parser_classes = (parsers.MultiPartParser, ) authentication_classes = BaseViewSet.authentication_classes permission_classes = BaseViewSet.permission_classes + renderer_classes = BaseViewSet.renderer_classes + parser_classes = (MultiPartParser, ) serializer_class = CollectionItemChunkSerializer lookup_field = 'uid' @@ -602,6 +609,8 @@ class InvitationIncomingViewSet(InvitationBaseViewSet): class AuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] authentication_classes = BaseViewSet.authentication_classes + renderer_classes = BaseViewSet.renderer_classes + parser_classes = BaseViewSet.parser_classes def get_encryption_key(self, salt): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) @@ -757,6 +766,8 @@ class AuthenticationViewSet(viewsets.ViewSet): class TestAuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] + renderer_classes = BaseViewSet.renderer_classes + parser_classes = BaseViewSet.parser_classes def list(self, request): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) From 3dfceb63b139efbf5d7196bf77e5a78d65761fac Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 29 Jun 2020 14:50:06 +0300 Subject: [PATCH 225/511] Views: move the base64 encoding to the renderers. Hard-coding the serialization encoding in the serializers is wrong. This fix now enables us to change to easily change to msgpack as the transport layer. --- django_etebase/renderers.py | 18 ++++++++++++++++++ django_etebase/serializers.py | 21 +++++++++++---------- django_etebase/views.py | 12 ++++++------ 3 files changed, 35 insertions(+), 16 deletions(-) create mode 100644 django_etebase/renderers.py diff --git a/django_etebase/renderers.py b/django_etebase/renderers.py new file mode 100644 index 0000000..43c1a0d --- /dev/null +++ b/django_etebase/renderers.py @@ -0,0 +1,18 @@ +from rest_framework.utils.encoders import JSONEncoder as DRFJSONEncoder +from rest_framework.renderers import JSONRenderer as DRFJSONRenderer + +from .serializers import b64encode + + +class JSONEncoder(DRFJSONEncoder): + def default(self, obj): + if isinstance(obj, bytes) or isinstance(obj, memoryview): + return b64encode(obj) + return super().default(obj) + + +class JSONRenderer(DRFJSONRenderer): + """ + Renderer which serializes to JSON with support for our base64 + """ + encoder_class = JSONEncoder diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 1876572..f78cd6b 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -56,18 +56,19 @@ def b64decode(data): return base64.urlsafe_b64decode(data) +def b64decode_or_bytes(data): + if isinstance(data, bytes): + return data + else: + return b64decode(data) + + class BinaryBase64Field(serializers.Field): def to_representation(self, value): - if self.context.get('supports_binary', False): - return value - else: - return b64encode(value) + return value def to_internal_value(self, data): - if isinstance(data, bytes): - return data - else: - return b64decode(data) + return b64decode_or_bytes(data) class CollectionEncryptionKeyField(BinaryBase64Field): @@ -91,14 +92,14 @@ class ChunksField(serializers.RelatedField): obj = obj.chunk if self.context.get('prefetch'): with open(obj.chunkFile.path, 'rb') as f: - return (obj.uid, b64encode(f.read())) + return (obj.uid, f.read()) else: return (obj.uid, ) def to_internal_value(self, data): if data[0] is None or data[1] is None: raise serializers.ValidationError('null is not allowed') - return (data[0], b64decode(data[1])) + return (data[0], b64decode_or_bytes(data[1])) class CollectionItemChunkSerializer(serializers.ModelSerializer): diff --git a/django_etebase/views.py b/django_etebase/views.py index 93a7645..970ad8c 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -29,7 +29,7 @@ from rest_framework import viewsets from rest_framework.decorators import action as action_decorator from rest_framework.response import Response from rest_framework.parsers import JSONParser, FormParser, MultiPartParser -from rest_framework.renderers import JSONRenderer, BrowsableAPIRenderer +from rest_framework.renderers import BrowsableAPIRenderer import nacl.encoding import nacl.signing @@ -42,6 +42,7 @@ from .drf_msgpack.parsers import MessagePackParser from .drf_msgpack.renderers import MessagePackRenderer from . import app_settings, permissions +from .renderers import JSONRenderer from .models import ( Collection, CollectionItem, @@ -53,7 +54,6 @@ from .models import ( UserInfo, ) from .serializers import ( - b64encode, AuthenticationChangePasswordInnerSerializer, AuthenticationSignupSerializer, AuthenticationLoginChallengeSerializer, @@ -700,8 +700,8 @@ class AuthenticationViewSet(viewsets.ViewSet): challenge = box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder) ret = { - "salt": b64encode(salt), - "challenge": b64encode(challenge), + "salt": salt, + "challenge": challenge, "version": user.userinfo.version, } return Response(ret, status=status.HTTP_200_OK) @@ -717,7 +717,7 @@ class AuthenticationViewSet(viewsets.ViewSet): response = msgpack_decode(response_raw) signature = outer_serializer.validated_data['signature'] - context = {'host': request.get_host(), 'supports_binary': True} + context = {'host': request.get_host()} serializer = AuthenticationLoginInnerSerializer(data=response, context=context) serializer.is_valid(raise_exception=True) @@ -750,7 +750,7 @@ class AuthenticationViewSet(viewsets.ViewSet): response = msgpack_decode(response_raw) signature = outer_serializer.validated_data['signature'] - context = {'host': request.get_host(), 'supports_binary': True} + context = {'host': request.get_host()} serializer = AuthenticationChangePasswordInnerSerializer(request.user.userinfo, data=response, context=context) serializer.is_valid(raise_exception=True) From f147f4ae58561b89cbac32d70e937e7e5d41ad83 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 29 Jun 2020 15:31:29 +0300 Subject: [PATCH 226/511] Serializers: allow encryptionKey to be null. --- django_etebase/serializers.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index f78cd6b..f2da771 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -122,7 +122,7 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): class CollectionItemSerializer(serializers.ModelSerializer): - encryptionKey = BinaryBase64Field(required=False, default=None) + encryptionKey = BinaryBase64Field(required=False, default=None, allow_null=True) etag = serializers.CharField(allow_null=True, write_only=True) content = CollectionItemRevisionSerializer(many=False) @@ -192,7 +192,7 @@ class CollectionSerializer(serializers.ModelSerializer): stoken = serializers.CharField(read_only=True) uid = serializers.CharField(source='main_item.uid') - encryptionKey = BinaryBase64Field(source='main_item.encryptionKey', required=False, default=None) + encryptionKey = BinaryBase64Field(source='main_item.encryptionKey', required=False, default=None, allow_null=True) etag = serializers.CharField(allow_null=True, write_only=True) version = serializers.IntegerField(min_value=0, source='main_item.version') content = CollectionItemRevisionSerializer(many=False, source='main_item.content') From f69c3a327cbb80a058d9fd60b02afe81f65bfa12 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 5 Jul 2020 13:15:42 +0300 Subject: [PATCH 227/511] Revert "django_etebase: make migration generic and not depend on myauth." This reverts commit 925dcac0fb99204e3373251e12f8496721879361. --- django_etebase/migrations/0002_userinfo.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/migrations/0002_userinfo.py b/django_etebase/migrations/0002_userinfo.py index bfeb2cf..6da0bb8 100644 --- a/django_etebase/migrations/0002_userinfo.py +++ b/django_etebase/migrations/0002_userinfo.py @@ -8,7 +8,7 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - # XXX removed this to make this migration generic ('myauth', '0001_initial'), + ('myauth', '0001_initial'), ('django_etebase', '0001_initial'), ] From 4aa3daaa97d62fb6b826374a0b17a64383b743b1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 5 Jul 2020 14:57:38 +0300 Subject: [PATCH 228/511] Create a new django project. --- etebase_server/__init__.py | 0 etebase_server/asgi.py | 16 +++++ etebase_server/settings.py | 120 +++++++++++++++++++++++++++++++++++++ etebase_server/urls.py | 21 +++++++ etebase_server/wsgi.py | 16 +++++ manage.py | 21 +++++++ 6 files changed, 194 insertions(+) create mode 100644 etebase_server/__init__.py create mode 100644 etebase_server/asgi.py create mode 100644 etebase_server/settings.py create mode 100644 etebase_server/urls.py create mode 100644 etebase_server/wsgi.py create mode 100755 manage.py diff --git a/etebase_server/__init__.py b/etebase_server/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/etebase_server/asgi.py b/etebase_server/asgi.py new file mode 100644 index 0000000..44f1c53 --- /dev/null +++ b/etebase_server/asgi.py @@ -0,0 +1,16 @@ +""" +ASGI config for etebase_server project. + +It exposes the ASGI callable as a module-level variable named ``application``. + +For more information on this file, see +https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/ +""" + +import os + +from django.core.asgi import get_asgi_application + +os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'etebase_server.settings') + +application = get_asgi_application() diff --git a/etebase_server/settings.py b/etebase_server/settings.py new file mode 100644 index 0000000..4ec2216 --- /dev/null +++ b/etebase_server/settings.py @@ -0,0 +1,120 @@ +""" +Django settings for etebase_server project. + +Generated by 'django-admin startproject' using Django 3.0.3. + +For more information on this file, see +https://docs.djangoproject.com/en/3.0/topics/settings/ + +For the full list of settings and their values, see +https://docs.djangoproject.com/en/3.0/ref/settings/ +""" + +import os + +# Build paths inside the project like this: os.path.join(BASE_DIR, ...) +BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + +# Quick-start development settings - unsuitable for production +# See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/ + +# SECURITY WARNING: keep the secret key used in production secret! +# SECRET_KEY = '' + +# SECURITY WARNING: don't run with debug turned on in production! +DEBUG = True + +ALLOWED_HOSTS = [] + + +# Application definition + +INSTALLED_APPS = [ + 'django.contrib.admin', + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', +] + +MIDDLEWARE = [ + 'django.middleware.security.SecurityMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.common.CommonMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', + 'django.middleware.clickjacking.XFrameOptionsMiddleware', +] + +ROOT_URLCONF = 'etebase_server.urls' + +TEMPLATES = [ + { + 'BACKEND': 'django.template.backends.django.DjangoTemplates', + 'DIRS': [], + 'APP_DIRS': True, + 'OPTIONS': { + 'context_processors': [ + 'django.template.context_processors.debug', + 'django.template.context_processors.request', + 'django.contrib.auth.context_processors.auth', + 'django.contrib.messages.context_processors.messages', + ], + }, + }, +] + +WSGI_APPLICATION = 'etebase_server.wsgi.application' + + +# Database +# https://docs.djangoproject.com/en/3.0/ref/settings/#databases + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), + } +} + + +# Password validation +# https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators + +AUTH_PASSWORD_VALIDATORS = [ + { + 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', + }, + { + 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', + }, + { + 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', + }, + { + 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', + }, +] + + +# Internationalization +# https://docs.djangoproject.com/en/3.0/topics/i18n/ + +LANGUAGE_CODE = 'en-us' + +TIME_ZONE = 'UTC' + +USE_I18N = True + +USE_L10N = True + +USE_TZ = True + + +# Static files (CSS, JavaScript, Images) +# https://docs.djangoproject.com/en/3.0/howto/static-files/ + +STATIC_URL = '/static/' diff --git a/etebase_server/urls.py b/etebase_server/urls.py new file mode 100644 index 0000000..4ac11e2 --- /dev/null +++ b/etebase_server/urls.py @@ -0,0 +1,21 @@ +"""etebase_server URL Configuration + +The `urlpatterns` list routes URLs to views. For more information please see: + https://docs.djangoproject.com/en/3.0/topics/http/urls/ +Examples: +Function views + 1. Add an import: from my_app import views + 2. Add a URL to urlpatterns: path('', views.home, name='home') +Class-based views + 1. Add an import: from other_app.views import Home + 2. Add a URL to urlpatterns: path('', Home.as_view(), name='home') +Including another URLconf + 1. Import the include() function: from django.urls import include, path + 2. Add a URL to urlpatterns: path('blog/', include('blog.urls')) +""" +from django.contrib import admin +from django.urls import path + +urlpatterns = [ + path('admin/', admin.site.urls), +] diff --git a/etebase_server/wsgi.py b/etebase_server/wsgi.py new file mode 100644 index 0000000..cf449a1 --- /dev/null +++ b/etebase_server/wsgi.py @@ -0,0 +1,16 @@ +""" +WSGI config for etebase_server project. + +It exposes the WSGI callable as a module-level variable named ``application``. + +For more information on this file, see +https://docs.djangoproject.com/en/3.0/howto/deployment/wsgi/ +""" + +import os + +from django.core.wsgi import get_wsgi_application + +os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'etebase_server.settings') + +application = get_wsgi_application() diff --git a/manage.py b/manage.py new file mode 100755 index 0000000..b793fd2 --- /dev/null +++ b/manage.py @@ -0,0 +1,21 @@ +#!/usr/bin/env python +"""Django's command-line utility for administrative tasks.""" +import os +import sys + + +def main(): + os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'etebase_server.settings') + try: + from django.core.management import execute_from_command_line + except ImportError as exc: + raise ImportError( + "Couldn't import Django. Are you sure it's installed and " + "available on your PYTHONPATH environment variable? Did you " + "forget to activate a virtual environment?" + ) from exc + execute_from_command_line(sys.argv) + + +if __name__ == '__main__': + main() From 08c4aa9d43b31c2004433538cbb719f1eb21e5bc Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 5 Jul 2020 15:09:46 +0300 Subject: [PATCH 229/511] Add .gitignore. --- .gitignore | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7f220af --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +/journal +/db.sqlite3* +Session.vim +/.venv +/assets +/logs +/.coverage +/tmp +/media + +__pycache__ +.*.swp + + +/etebase_server_settings.py From cc163d27af354052435047046e5107fd281f008c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 5 Jul 2020 15:04:24 +0300 Subject: [PATCH 230/511] Add settings and configuration to run the etebase app. --- etebase_server/settings.py | 37 ++++++++++++++++++++++++++++++++----- etebase_server/urls.py | 23 +++++++---------------- templates/success.html | 12 ++++++++++++ 3 files changed, 51 insertions(+), 21 deletions(-) create mode 100644 templates/success.html diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 4ec2216..94ca4d2 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -15,15 +15,17 @@ import os # Build paths inside the project like this: os.path.join(BASE_DIR, ...) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +AUTH_USER_MODEL = 'myauth.User' + # Quick-start development settings - unsuitable for production # See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/ -# SECURITY WARNING: keep the secret key used in production secret! -# SECRET_KEY = '' +# Should be set in the site specific settings +# SECRET_KEY = # SECURITY WARNING: don't run with debug turned on in production! -DEBUG = True +DEBUG = False ALLOWED_HOSTS = [] @@ -37,11 +39,18 @@ INSTALLED_APPS = [ 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', + 'corsheaders', + 'rest_framework', + 'fullurl', + 'myauth.apps.MyauthConfig', + 'django_etebase.apps.DjangoEtebaseConfig', + 'django_etebase.token_auth.apps.TokenAuthConfig', ] MIDDLEWARE = [ 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', + 'corsheaders.middleware.CorsMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', @@ -54,7 +63,9 @@ ROOT_URLCONF = 'etebase_server.urls' TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', - 'DIRS': [], + 'DIRS': [ + os.path.join(BASE_DIR, 'templates') + ], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ @@ -76,7 +87,8 @@ WSGI_APPLICATION = 'etebase_server.wsgi.application' DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), + 'NAME': os.environ.get('ETEBASE_DB_PATH', + os.path.join(BASE_DIR, 'db.sqlite3')), } } @@ -113,8 +125,23 @@ USE_L10N = True USE_TZ = True +# Cors +CORS_ORIGIN_ALLOW_ALL = True # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/3.0/howto/static-files/ +STATIC_ROOT = os.environ.get('DJANGO_STATIC_ROOT', os.path.join(BASE_DIR, 'assets')) STATIC_URL = '/static/' + +MEDIA_ROOT = os.environ.get('DJANGO_MEDIA_ROOT', os.path.join(BASE_DIR, 'media')) +MEDIA_URL = '/user-media/' + +ETEBASE_API_PERMISSIONS = ('rest_framework.permissions.IsAuthenticated', ) +ETEBASE_API_AUTHENTICATORS = ('django_etebase.token_auth.authentication.TokenAuthentication', + 'rest_framework.authentication.SessionAuthentication') + +try: + from etebase_server_settings import * +except ImportError: + pass diff --git a/etebase_server/urls.py b/etebase_server/urls.py index 4ac11e2..0c114af 100644 --- a/etebase_server/urls.py +++ b/etebase_server/urls.py @@ -1,21 +1,12 @@ -"""etebase_server URL Configuration - -The `urlpatterns` list routes URLs to views. For more information please see: - https://docs.djangoproject.com/en/3.0/topics/http/urls/ -Examples: -Function views - 1. Add an import: from my_app import views - 2. Add a URL to urlpatterns: path('', views.home, name='home') -Class-based views - 1. Add an import: from other_app.views import Home - 2. Add a URL to urlpatterns: path('', Home.as_view(), name='home') -Including another URLconf - 1. Import the include() function: from django.urls import include, path - 2. Add a URL to urlpatterns: path('blog/', include('blog.urls')) -""" +from django.conf.urls import include, url from django.contrib import admin from django.urls import path +from django.views.generic import TemplateView urlpatterns = [ - path('admin/', admin.site.urls), + url(r'^api/', include('django_etebase.urls')), + url(r'^admin/', admin.site.urls), + url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), + + path('', TemplateView.as_view(template_name='success.html')), ] diff --git a/templates/success.html b/templates/success.html new file mode 100644 index 0000000..c7cf494 --- /dev/null +++ b/templates/success.html @@ -0,0 +1,12 @@ + + + + It works! + + +

It works!

+

+ Please refer to the README to complete the final steps if you haven't done so already. +

+ + From ee23707fffee4347e0fea3c3e97ef3e601309ca0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 5 Jul 2020 15:43:37 +0300 Subject: [PATCH 231/511] Debug reset: put the whole request in a transaction. --- django_etebase/views.py | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 970ad8c..a8a6100 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -778,23 +778,24 @@ class TestAuthenticationViewSet(viewsets.ViewSet): if not settings.DEBUG: return HttpResponseBadRequest("Only allowed in debug mode.") - user = get_object_or_404(User.objects.all(), username=request.data.get('user').get('username')) + with transaction.atomic(): + user = get_object_or_404(User.objects.all(), username=request.data.get('user').get('username')) - # Only allow test users for extra safety - if not getattr(user, User.USERNAME_FIELD).startswith('test_user'): - return HttpResponseBadRequest("Endpoint not allowed for user.") + # Only allow test users for extra safety + if not getattr(user, User.USERNAME_FIELD).startswith('test_user'): + return HttpResponseBadRequest("Endpoint not allowed for user.") - if hasattr(user, 'userinfo'): - user.userinfo.delete() + if hasattr(user, 'userinfo'): + user.userinfo.delete() - serializer = AuthenticationSignupSerializer(data=request.data) - serializer.is_valid(raise_exception=True) - serializer.save() + serializer = AuthenticationSignupSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + serializer.save() - # Delete all of the journal data for this user for a clear test env - user.collection_set.all().delete() - user.incoming_invitations.all().delete() + # Delete all of the journal data for this user for a clear test env + user.collection_set.all().delete() + user.incoming_invitations.all().delete() - # FIXME: also delete chunk files!!! + # FIXME: also delete chunk files!!! return HttpResponse() From 2d4410ef36c5376de0b8e81b5e931157b3327e38 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 8 Jul 2020 17:45:44 +0300 Subject: [PATCH 232/511] Add license file. --- LICENSE | 661 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 661 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..be3f7b2 --- /dev/null +++ b/LICENSE @@ -0,0 +1,661 @@ + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. From e8e859fa6acccbefb65a73a706e47e6bbb280e23 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 8 Jul 2020 17:45:50 +0300 Subject: [PATCH 233/511] Add README. --- README.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..781fd92 --- /dev/null +++ b/README.md @@ -0,0 +1,113 @@ +

+ +

EteSync - Secure Data Sync

+

+ +A skeleton app for running your own [Etebase](https://www.etebase.com) server + +# Installation + +## From source + +Before installing the EteSync server make sure you install `virtualenv` (for **Python 3**): + +* Arch Linux: `pacman -S python-virtualenv` +* Debian/Ubuntu: `apt-get install python3-virtualenv` +* Mac/Windows/Other Linux: install virtualenv or just skip the instructions mentioning virtualenv. + +Then just clone the git repo and set up this app: + +``` +git clone https://github.com/etesync/server-skeleton.git + +cd server-skeleton + +# Set up the environment and deps +virtualenv -p python3 venv # If doesn't work, try: virtualenv3 venv +source venv/bin/activate + +pip install -r requirements.txt +``` + +# Configuration + +If you are familiar with Django you can just edit the [settings file](etesync_server/settings.py) +according to the [Django deployment checklist](https://docs.djangoproject.com/en/dev/howto/deployment/checklist/) +if you are not, we will soon provide a simple configuration file for easy deployment like we had with EteSync. + +Some particular settings that should be edited are: + * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) + -- this is the list of host/domain names or addresses on which the app +will be served + * [`DEBUG`](https://docs.djangoproject.com/en/1.11/ref/settings/#debug) + -- handy for debugging, set to `False` for production + * [`SECRET_KEY`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-SECRET_KEY) + -- an ephemeral secret used for various cryptographic signing and token +generation purposes. See below for how default configuration of +`SECRET_KEY` works for this project. + +Now you can initialise our django app + +``` +./manage.py migrate +``` + +And you are done! You can now run the debug server just to see everything works as expected by running: + +``` +./manage.py runserver 0.0.0.0:8000 +``` + +Using the debug server in production is not recommended, so please read the following section for a proper deployment. + +# Production deployment + +EteSync is based on Django so you should refer to one of the following + * The instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). + * Instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). + +There are more details about a proper production setup using uWSGI and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-uWSGI-and-Nginx). + +The webserver should also be configured to serve Etebase using TLS. +A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-EteSync) as well. + +# Usage + +Create yourself an admin user: + +``` +./manage.py createsuperuser +``` + +At this stage you can either just use the admin user, or better yet, go to: ```www.your-etesync-install.com/admin``` +and create a non-privileged user that you can use. + +That's it! + +Now all that's left is to open the EteSync app, add an account, and set your custom server address under the "advance" section. + +# `SECRET_KEY` and `secret.txt` + +The default configuration creates a file “`secret.txt`” in the project’s +base directory, which is used as the value of the Django `SECRET_KEY` +setting. You can revoke this key by deleting the `secret.txt` file and the +next time the app is run, a new one will be generated. Make sure you keep +the `secret.txt` file secret (don’t accidentally commit it to version +control, exclude it from your backups, etc.). If you want to change to a +more secure system for storing secrets, edit `etesync_server/settings.py` +and implement your own method for setting `SECRET_KEY` (remove the line +where it uses the `get_secret_from_file` function). Read the Django docs +for more information about the `SECRET_KEY` and its uses. + +# Updating + +First, run `git pull --rebase` to update this repository. +Then, inside the virtualenv: +1. Run `pip install -U -r requirements.txt` to update the dependencies. +2. Run `python manage.py migrate` to perform database migrations. + +You can now restart the server. + +# Supporting Etebase + +Please consider registering an account even if you self-host in order to support the development of Etebase, or help by spreading the word. From 86c5d711a6fef03a853d4e5c15e30cd38ffd71c4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 10 Jul 2020 09:09:11 +0300 Subject: [PATCH 234/511] Chunk upload: item.uid can never be None so use it directly. --- django_etebase/models.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index b4b04fc..397600e 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -82,8 +82,7 @@ def chunk_directory_path(instance, filename): item = instance.item col = item.collection user_id = col.owner.id - item_uid = item.uid or 'main' - return Path('user_{}'.format(user_id), col.uid, item_uid, instance.uid) + return Path('user_{}'.format(user_id), col.uid, item.uid, instance.uid) class CollectionItemChunk(models.Model): From fae15fe420e8cba63ecd4cbf04f95ef44e5cec5e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 10 Jul 2020 09:27:34 +0300 Subject: [PATCH 235/511] Views: clean up how we use serializers and remove integrity_errors catch-alls. The integrity errors were a bad relic from the EteSync sources and needed to be removed. --- django_etebase/views.py | 139 +++++++++++++++++----------------------- 1 file changed, 58 insertions(+), 81 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index a8a6100..c8a537f 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -197,16 +197,10 @@ class CollectionViewSet(BaseViewSet): def create(self, request, *args, **kwargs): serializer = self.get_serializer(data=request.data) - if serializer.is_valid(): - try: - serializer.save(owner=self.request.user) - except IntegrityError: - content = {'code': 'integrity_error'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - return Response({}, status=status.HTTP_201_CREATED) + serializer.is_valid(raise_exception=True) + serializer.save(owner=self.request.user) - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + return Response({}, status=status.HTTP_201_CREATED) def list(self, request): queryset = self.get_queryset() @@ -326,35 +320,33 @@ class CollectionItemViewSet(BaseViewSet): queryset = self.get_queryset() serializer = CollectionItemBulkGetSerializer(data=request.data, many=True) - if serializer.is_valid(): - # FIXME: make configurable? - item_limit = 200 - - if len(serializer.validated_data) > item_limit: - content = {'code': 'too_many_items', - 'detail': 'Request has too many items. Limit: {}'. format(item_limit)} - return Response(content, status=status.HTTP_400_BAD_REQUEST) + serializer.is_valid(raise_exception=True) + # FIXME: make configurable? + item_limit = 200 - queryset, stoken_rev = self.filter_by_stoken(request, queryset) + if len(serializer.validated_data) > item_limit: + content = {'code': 'too_many_items', + 'detail': 'Request has too many items. Limit: {}'. format(item_limit)} + return Response(content, status=status.HTTP_400_BAD_REQUEST) - uids, etags = zip(*[(item['uid'], item.get('etag')) for item in serializer.validated_data]) - revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) - queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) + queryset, stoken_rev = self.filter_by_stoken(request, queryset) - new_stoken = self.get_queryset_stoken(queryset) - stoken = stoken_rev and stoken_rev.uid - new_stoken = new_stoken or stoken + uids, etags = zip(*[(item['uid'], item.get('etag')) for item in serializer.validated_data]) + revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) + queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) - serializer = self.get_serializer(queryset, many=True) + new_stoken = self.get_queryset_stoken(queryset) + stoken = stoken_rev and stoken_rev.uid + new_stoken = new_stoken or stoken - ret = { - 'data': serializer.data, - 'stoken': new_stoken, - 'done': True, # we always return all the items, so it's always done - } - return Response(ret) + serializer = self.get_serializer(queryset, many=True) - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + ret = { + 'data': serializer.data, + 'stoken': new_stoken, + 'done': True, # we always return all the items, so it's always done + } + return Response(ret) @action_decorator(detail=False, methods=['POST']) def batch(self, request, collection_uid=None): @@ -383,12 +375,7 @@ class CollectionItemViewSet(BaseViewSet): ser_valid = serializer.is_valid() deps_ser_valid = (deps is None or deps_serializer.is_valid()) if ser_valid and deps_ser_valid: - try: - items = serializer.save(collection=collection_object) - except IntegrityError: - # FIXME: return the items with a bad token (including deps) so we don't have to fetch them after - content = {'code': 'integrity_error'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) + items = serializer.save(collection=collection_object) ret = { } @@ -423,16 +410,10 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): col_it = get_object_or_404(col.items, uid=collection_item_uid) serializer = self.get_serializer_class()(data=request.data) - if serializer.is_valid(): - try: - serializer.save(item=col_it) - except IntegrityError: - content = {'code': 'integrity_error'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - return Response({}, status=status.HTTP_201_CREATED) + serializer.is_valid(raise_exception=True) + serializer.save(item=col_it) - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + return Response({}, status=status.HTTP_201_CREATED) @action_decorator(detail=True, methods=['GET']) def download(self, request, collection_uid=None, collection_item_uid=None, uid=None): @@ -559,22 +540,20 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): def create(self, request, *args, **kwargs): serializer = self.get_serializer(data=request.data) - if serializer.is_valid(): - collection_uid = serializer.validated_data.get('collection', {}).get('uid') - - try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) - except Collection.DoesNotExist: - raise Http404('Collection does not exist') + serializer.is_valid(raise_exception=True) + collection_uid = serializer.validated_data.get('collection', {}).get('uid') - if not permissions.is_collection_admin(collection, request.user): - raise PermissionDenied('User is not an admin of this collection') + try: + collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) + except Collection.DoesNotExist: + raise Http404('Collection does not exist') - serializer.save(collection=collection) + if not permissions.is_collection_admin(collection, request.user): + raise PermissionDenied('User is not an admin of this collection') - return Response({}, status=status.HTTP_201_CREATED) + serializer.save(collection=collection) - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + return Response({}, status=status.HTTP_201_CREATED) @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) def fetch_user_profile(self, request): @@ -685,28 +664,26 @@ class AuthenticationViewSet(viewsets.ViewSet): from datetime import datetime serializer = AuthenticationLoginChallengeSerializer(data=request.data) - if serializer.is_valid(): - username = serializer.validated_data.get('username') - user = self.get_login_user(username) - - salt = bytes(user.userinfo.salt) - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = { - "timestamp": int(datetime.now().timestamp()), - "userId": user.id, - } - challenge = box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder) - - ret = { - "salt": salt, - "challenge": challenge, - "version": user.userinfo.version, - } - return Response(ret, status=status.HTTP_200_OK) - - return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) + serializer.is_valid(raise_exception=True) + username = serializer.validated_data.get('username') + user = self.get_login_user(username) + + salt = bytes(user.userinfo.salt) + enc_key = self.get_encryption_key(salt) + box = nacl.secret.SecretBox(enc_key) + + challenge_data = { + "timestamp": int(datetime.now().timestamp()), + "userId": user.id, + } + challenge = box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder) + + ret = { + "salt": salt, + "challenge": challenge, + "version": user.userinfo.version, + } + return Response(ret, status=status.HTTP_200_OK) @action_decorator(detail=False, methods=['POST']) def login(self, request): From 9a518b3907ffd0ba95ead85724d2674988d46fcb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 10 Jul 2020 09:29:19 +0300 Subject: [PATCH 236/511] Chunks: add error handling for chunks having content or not existing. If the chunk already has a content and we try to upload it again, we assume the previous content was correct and this one is the same (chunks are immutable). We can't actually ensure they are the same due to the encryption, though they should be. If a chunk is being uploaded for the first time and doesn't have a content, throw a validation error rather than throwing an ugly error. --- django_etebase/serializers.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index f2da771..3f1b084 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -29,15 +29,19 @@ def process_revisions_for_item(item, revision_data): chunks = revision_data.pop('chunks_relation') for chunk in chunks: uid = chunk[0] + chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() if len(chunk) > 1: content = chunk[1] - chunk = models.CollectionItemChunk(uid=uid, item=item) - chunk.chunkFile.save('IGNORED', ContentFile(content)) - chunk.save() - chunks_objs.append(chunk) + # If the chunk already exists we assume it's fine. Otherwise, we upload it. + if chunk_obj is None: + chunk_obj = models.CollectionItemChunk(uid=uid, item=item) + chunk_obj.chunkFile.save('IGNORED', ContentFile(content)) + chunk_obj.save() else: - chunk = models.CollectionItemChunk.objects.get(uid=uid) - chunks_objs.append(chunk) + if chunk_obj is None: + raise serializers.ValidationError('Tried to create a new chunk without content') + + chunks_objs.append(chunk_obj) stoken = models.Stoken.objects.create() From 7ec45434ba4c29a6d8225dc659bab8be54797e8d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 12 Jul 2020 11:11:33 +0300 Subject: [PATCH 237/511] User: make username case insensitive (and save original styling). We want 'User' and 'UsEr' to mean the same user. Apparently that's not the default in django. This normalizes the user to ensure we enforce this. --- django_etebase/serializers.py | 7 ++++++- myauth/models.py | 13 ++++++++++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 3f1b084..908f126 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -382,7 +382,12 @@ class AuthenticationSignupSerializer(serializers.Serializer): user_data = validated_data.pop('user') with transaction.atomic(): - instance, _ = User.objects.get_or_create(**user_data) + try: + instance = User.objects.get_by_natural_key(user_data['username']) + except User.DoesNotExist: + # Create the user and save the casing the user chose as the first name + instance = User.objects.create_user(**user_data, first_name=user_data['username']) + if hasattr(instance, 'userinfo'): raise serializers.ValidationError('User already exists') diff --git a/myauth/models.py b/myauth/models.py index 4afc27c..4046b2f 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -1,4 +1,4 @@ -from django.contrib.auth.models import AbstractUser +from django.contrib.auth.models import AbstractUser, UserManager as DjangoUserManager from django.core import validators from django.db import models from django.utils.deconstruct import deconstructible @@ -15,9 +15,16 @@ class UnicodeUsernameValidator(validators.RegexValidator): flags = 0 +class UserManager(DjangoUserManager): + def get_by_natural_key(self, username): + return self.get(**{self.model.USERNAME_FIELD + '__iexact': username}) + + class User(AbstractUser): username_validator = UnicodeUsernameValidator() + objects = UserManager() + username = models.CharField( _('username'), max_length=150, @@ -28,3 +35,7 @@ class User(AbstractUser): 'unique': _("A user with that username already exists."), }, ) + + @classmethod + def normalize_username(cls, username): + return super().normalize_username(username).lower() From 9f1bfceda7b76dcc829b9ddc639494cb65297259 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 12 Jul 2020 11:27:47 +0300 Subject: [PATCH 238/511] Increase token ttl to 30 days. --- django_etebase/token_auth/models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/token_auth/models.py b/django_etebase/token_auth/models.py index 9ac0955..0fe4766 100644 --- a/django_etebase/token_auth/models.py +++ b/django_etebase/token_auth/models.py @@ -11,7 +11,7 @@ def generate_key(): def get_default_expiry(): - return timezone.now() + timezone.timedelta(days=14) + return timezone.now() + timezone.timedelta(days=30) class AuthToken(models.Model): From 41a03e9d3bbd2a7345a93b4ba6ded66af5a60d64 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 12 Jul 2020 13:23:45 +0300 Subject: [PATCH 239/511] Invitation: fix the checks making sure you can't invite yourself. --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 908f126..7fe5050 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -294,7 +294,7 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): def validate_user(self, value): request = self.context['request'] - if request.user == value.lower(): + if request.user.username == value.lower(): raise serializers.ValidationError('Inviting yourself is not allowed') return value From 9ea01d4d938ba90c99adfe4c492f9c8cdf4cd60d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 11:15:42 +0300 Subject: [PATCH 240/511] CollectionMemberSerializer: change the user field to be read only. --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 7fe5050..13199b3 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -255,7 +255,7 @@ class CollectionMemberSerializer(serializers.ModelSerializer): username = serializers.SlugRelatedField( source='user', slug_field=User.USERNAME_FIELD, - queryset=User.objects + read_only=True, ) class Meta: From 3680bd53b1068d9ccb425d3b953dcdf08640af3f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 14:26:39 +0300 Subject: [PATCH 241/511] Views: change according to DRF best practices. --- django_etebase/views.py | 56 ++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index c8a537f..7e8bf98 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -153,7 +153,7 @@ class BaseViewSet(viewsets.ModelViewSet): return result, new_stoken, done # Change how our list works by default - def list(self, request, collection_uid=None): + def list(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset() serializer = self.get_serializer(queryset, many=True) @@ -185,11 +185,11 @@ class CollectionViewSet(BaseViewSet): context.update({'request': self.request, 'prefetch': prefetch}) return context - def destroy(self, request, uid=None): + def destroy(self, request, uid=None, *args, **kwargs): # FIXME: implement return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - def partial_update(self, request, uid=None): + def partial_update(self, request, uid=None, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) def update(self, request, *args, **kwargs): @@ -202,7 +202,7 @@ class CollectionViewSet(BaseViewSet): return Response({}, status=status.HTTP_201_CREATED) - def list(self, request): + def list(self, request, *args, **kwargs): queryset = self.get_queryset() result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) @@ -251,21 +251,21 @@ class CollectionItemViewSet(BaseViewSet): context.update({'request': self.request, 'prefetch': prefetch}) return context - def create(self, request, collection_uid=None): + def create(self, request, collection_uid=None, *args, **kwargs): # We create using batch and transaction return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - def destroy(self, request, collection_uid=None, uid=None): + def destroy(self, request, collection_uid=None, uid=None, *args, **kwargs): # We can't have destroy because we need to get data from the user (in the body) such as hmac. return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - def update(self, request, collection_uid=None, uid=None): + def update(self, request, collection_uid=None, uid=None, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - def partial_update(self, request, collection_uid=None, uid=None): + def partial_update(self, request, collection_uid=None, uid=None, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - def list(self, request, collection_uid=None): + def list(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset() if not self.request.query_params.get('withCollection', False): @@ -283,7 +283,7 @@ class CollectionItemViewSet(BaseViewSet): return Response(ret) @action_decorator(detail=True, methods=['GET']) - def revision(self, request, collection_uid=None, uid=None): + def revision(self, request, collection_uid=None, uid=None, *args, **kwargs): col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) item = get_object_or_404(col.items, uid=uid) @@ -316,7 +316,7 @@ class CollectionItemViewSet(BaseViewSet): # FIXME: rename to something consistent with what the clients have - maybe list_updates? @action_decorator(detail=False, methods=['POST']) - def fetch_updates(self, request, collection_uid=None): + def fetch_updates(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset() serializer = CollectionItemBulkGetSerializer(data=request.data, many=True) @@ -349,11 +349,11 @@ class CollectionItemViewSet(BaseViewSet): return Response(ret) @action_decorator(detail=False, methods=['POST']) - def batch(self, request, collection_uid=None): + def batch(self, request, collection_uid=None, *args, **kwargs): return self.transaction(request, collection_uid, validate_etag=False) @action_decorator(detail=False, methods=['POST']) - def transaction(self, request, collection_uid=None, validate_etag=True): + def transaction(self, request, collection_uid=None, validate_etag=True, *args, **kwargs): stoken = request.GET.get('stoken', None) with transaction.atomic(): # We need this for locking on the collection object collection_object = get_object_or_404( @@ -405,7 +405,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): user = self.request.user return queryset.filter(members__user=user) - def create(self, request, collection_uid=None, collection_item_uid=None): + def create(self, request, collection_uid=None, collection_item_uid=None, *args, **kwargs): col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) @@ -416,7 +416,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_201_CREATED) @action_decorator(detail=True, methods=['GET']) - def download(self, request, collection_uid=None, collection_item_uid=None, uid=None): + def download(self, request, collection_uid=None, collection_item_uid=None, uid=None, *args, **kwargs): import os from django.views.static import serve @@ -461,7 +461,7 @@ class CollectionMemberViewSet(BaseViewSet): def get_stoken_obj_id(self, request): return request.GET.get('iterator', None) - def list(self, request, collection_uid=None): + def list(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset().order_by('id') result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) serializer = self.get_serializer(result, many=True) @@ -474,7 +474,7 @@ class CollectionMemberViewSet(BaseViewSet): return Response(ret) - def create(self, request): + def create(self, request, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) # FIXME: block leaving if we are the last admins - should be deleted / assigned in this case depending if there @@ -483,7 +483,7 @@ class CollectionMemberViewSet(BaseViewSet): instance.revoke() @action_decorator(detail=False, methods=['POST'], permission_classes=our_base_permission_classes) - def leave(self, request, collection_uid=None): + def leave(self, request, collection_uid=None, *args, **kwargs): collection_uid = self.kwargs['collection_uid'] col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) @@ -499,7 +499,7 @@ class InvitationBaseViewSet(BaseViewSet): lookup_field = 'uid' lookup_url_kwarg = 'invitation_uid' - def list(self, request, collection_uid=None): + def list(self, request, collection_uid=None, *args, **kwargs): limit = int(request.GET.get('limit', 50)) iterator = request.GET.get('iterator', None) @@ -556,7 +556,7 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): return Response({}, status=status.HTTP_201_CREATED) @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) - def fetch_user_profile(self, request): + def fetch_user_profile(self, request, *args, **kwargs): username = request.GET.get('username') kwargs = {'owner__' + User.USERNAME_FIELD: username} user_info = get_object_or_404(UserInfo.objects.all(), **kwargs) @@ -574,7 +574,7 @@ class InvitationIncomingViewSet(InvitationBaseViewSet): return queryset.filter(user=self.request.user) @action_decorator(detail=True, allowed_methods=['POST'], methods=['POST']) - def accept(self, request, invitation_uid=None): + def accept(self, request, invitation_uid=None, *args, **kwargs): invitation = get_object_or_404(self.get_queryset(), uid=invitation_uid) context = self.get_serializer_context() context.update({'invitation': invitation}) @@ -605,11 +605,11 @@ class AuthenticationViewSet(viewsets.ViewSet): 'user': UserSerializer(user).data, } - def list(self, request): + def list(self, request, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) @action_decorator(detail=False, methods=['POST']) - def signup(self, request): + def signup(self, request, *args, **kwargs): serializer = AuthenticationSignupSerializer(data=request.data) serializer.is_valid(raise_exception=True) user = serializer.save() @@ -660,7 +660,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return None @action_decorator(detail=False, methods=['POST']) - def login_challenge(self, request): + def login_challenge(self, request, *args, **kwargs): from datetime import datetime serializer = AuthenticationLoginChallengeSerializer(data=request.data) @@ -686,7 +686,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(ret, status=status.HTTP_200_OK) @action_decorator(detail=False, methods=['POST']) - def login(self, request): + def login(self, request, *args, **kwargs): outer_serializer = AuthenticationLoginSerializer(data=request.data) outer_serializer.is_valid(raise_exception=True) @@ -713,13 +713,13 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(data, status=status.HTTP_200_OK) @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) - def logout(self, request): + def logout(self, request, *args, **kwargs): request.auth.delete() user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) return Response(status=status.HTTP_204_NO_CONTENT) @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) - def change_password(self, request): + def change_password(self, request, *args, **kwargs): outer_serializer = AuthenticationLoginSerializer(data=request.data) outer_serializer.is_valid(raise_exception=True) @@ -746,7 +746,7 @@ class TestAuthenticationViewSet(viewsets.ViewSet): renderer_classes = BaseViewSet.renderer_classes parser_classes = BaseViewSet.parser_classes - def list(self, request): + def list(self, request, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) @action_decorator(detail=False, methods=['POST']) From f9add36f18e04288a4b5ea7768666d70ebc1abc0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 14:30:18 +0300 Subject: [PATCH 242/511] Add support for custom user filtering. --- django_etebase/app_settings.py | 7 +++++++ django_etebase/serializers.py | 16 ++++++++++++---- django_etebase/utils.py | 12 ++++++++++++ django_etebase/views.py | 11 +++++++---- 4 files changed, 38 insertions(+), 8 deletions(-) create mode 100644 django_etebase/utils.py diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index b1fb4c3..7fe30b7 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -46,6 +46,13 @@ class AppSettings: ret.append(self.import_from_str(perm)) return ret + @property + def GET_USER_QUERYSET(self): # pylint: disable=invalid-name + get_user_queryset = self._setting("GET_USER_QUERYSET", None) + if get_user_queryset is not None: + return self.import_from_str(get_user_queryset) + return None + @property def CHALLENGE_VALID_SECONDS(self): # pylint: disable=invalid-name return self._setting("CHALLENGE_VALID_SECONDS", 60) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 13199b3..0655775 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -20,6 +20,7 @@ from django.contrib.auth import get_user_model from django.db import transaction from rest_framework import serializers from . import models +from .utils import get_user_queryset User = get_user_model() @@ -91,6 +92,15 @@ class CollectionContentField(BinaryBase64Field): return None +class UserSlugRelatedField(serializers.SlugRelatedField): + def get_queryset(self): + view = self.context.get('view', None) + return get_user_queryset(super().get_queryset(), view) + + def __init__(self, **kwargs): + super().__init__(slug_field=User.USERNAME_FIELD, **kwargs) + + class ChunksField(serializers.RelatedField): def to_representation(self, obj): obj = obj.chunk @@ -252,9 +262,8 @@ class CollectionSerializer(serializers.ModelSerializer): class CollectionMemberSerializer(serializers.ModelSerializer): - username = serializers.SlugRelatedField( + username = UserSlugRelatedField( source='user', - slug_field=User.USERNAME_FIELD, read_only=True, ) @@ -278,9 +287,8 @@ class CollectionMemberSerializer(serializers.ModelSerializer): class CollectionInvitationSerializer(serializers.ModelSerializer): - username = serializers.SlugRelatedField( + username = UserSlugRelatedField( source='user', - slug_field=User.USERNAME_FIELD, queryset=User.objects ) collection = serializers.CharField(source='collection.uid') diff --git a/django_etebase/utils.py b/django_etebase/utils.py new file mode 100644 index 0000000..315b82f --- /dev/null +++ b/django_etebase/utils.py @@ -0,0 +1,12 @@ +from django.contrib.auth import get_user_model +from . import app_settings + + +User = get_user_model() + + +def get_user_queryset(queryset, view): + custom_func = app_settings.GET_USER_QUERYSET + if custom_func is not None: + return custom_func(queryset, view) + return queryset diff --git a/django_etebase/views.py b/django_etebase/views.py index 7e8bf98..480843e 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -71,6 +71,7 @@ from .serializers import ( UserInfoPubkeySerializer, UserSerializer, ) +from .utils import get_user_queryset User = get_user_model() @@ -558,8 +559,9 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) def fetch_user_profile(self, request, *args, **kwargs): username = request.GET.get('username') - kwargs = {'owner__' + User.USERNAME_FIELD: username} - user_info = get_object_or_404(UserInfo.objects.all(), **kwargs) + kwargs = {User.USERNAME_FIELD: username} + user = get_object_or_404(get_user_queryset(User.objects.all(), self), **kwargs) + user_info = get_object_or_404(UserInfo.objects.all(), owner=user) serializer = UserInfoPubkeySerializer(user_info) return Response(serializer.data) @@ -597,7 +599,7 @@ class AuthenticationViewSet(viewsets.ViewSet): encoder=nacl.encoding.RawEncoder) def get_queryset(self): - return User.objects.all() + return get_user_queryset(User.objects.all(), self) def login_response_data(self, user): return { @@ -756,7 +758,8 @@ class TestAuthenticationViewSet(viewsets.ViewSet): return HttpResponseBadRequest("Only allowed in debug mode.") with transaction.atomic(): - user = get_object_or_404(User.objects.all(), username=request.data.get('user').get('username')) + user_queryset = get_user_queryset(User.objects.all(), self) + user = get_object_or_404(user_queryset, username=request.data.get('user').get('username')) # Only allow test users for extra safety if not getattr(user, User.USERNAME_FIELD).startswith('test_user'): From 5c2f4d96ad65441ae18f2a0c2e8faa0763eb5e10 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 14:35:31 +0300 Subject: [PATCH 243/511] app settings: cache all the properties rather than recalc every time. They never change during runtime anyway. --- django_etebase/app_settings.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index 7fe30b7..2b9da4a 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -11,6 +11,7 @@ # # You should have received a copy of the GNU General Public License # along with this program. If not, see . +from django.utils.functional import cached_property class AppSettings: @@ -29,7 +30,7 @@ class AppSettings: from django.conf import settings return getattr(settings, self.prefix + name, dflt) - @property + @cached_property def API_PERMISSIONS(self): # pylint: disable=invalid-name perms = self._setting("API_PERMISSIONS", ('rest_framework.permissions.IsAuthenticated', )) ret = [] @@ -37,7 +38,7 @@ class AppSettings: ret.append(self.import_from_str(perm)) return ret - @property + @cached_property def API_AUTHENTICATORS(self): # pylint: disable=invalid-name perms = self._setting("API_AUTHENTICATORS", ('rest_framework.authentication.TokenAuthentication', 'rest_framework.authentication.SessionAuthentication')) @@ -46,14 +47,14 @@ class AppSettings: ret.append(self.import_from_str(perm)) return ret - @property + @cached_property def GET_USER_QUERYSET(self): # pylint: disable=invalid-name get_user_queryset = self._setting("GET_USER_QUERYSET", None) if get_user_queryset is not None: return self.import_from_str(get_user_queryset) return None - @property + @cached_property def CHALLENGE_VALID_SECONDS(self): # pylint: disable=invalid-name return self._setting("CHALLENGE_VALID_SECONDS", 60) From a39617cf2e09e26b6209f203dd09715777556e26 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 15:26:05 +0300 Subject: [PATCH 244/511] Make sure usernames are case insensitive on lookup --- django_etebase/serializers.py | 3 +++ django_etebase/views.py | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 0655775..94ab3e7 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -100,6 +100,9 @@ class UserSlugRelatedField(serializers.SlugRelatedField): def __init__(self, **kwargs): super().__init__(slug_field=User.USERNAME_FIELD, **kwargs) + def to_internal_value(self, data): + return super().to_internal_value(data.lower()) + class ChunksField(serializers.RelatedField): def to_representation(self, obj): diff --git a/django_etebase/views.py b/django_etebase/views.py index 480843e..327bc08 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -439,7 +439,7 @@ class CollectionMemberViewSet(BaseViewSet): permission_classes = our_base_permission_classes + (permissions.IsCollectionAdmin, ) queryset = CollectionMember.objects.all() serializer_class = CollectionMemberSerializer - lookup_field = 'user__' + User.USERNAME_FIELD + lookup_field = f'user__{User.USERNAME_FIELD}__iexact' lookup_url_kwarg = 'username' stoken_id_fields = ['stoken__id'] @@ -559,7 +559,7 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) def fetch_user_profile(self, request, *args, **kwargs): username = request.GET.get('username') - kwargs = {User.USERNAME_FIELD: username} + kwargs = {User.USERNAME_FIELD: username.lower()} user = get_object_or_404(get_user_queryset(User.objects.all(), self), **kwargs) user_info = get_object_or_404(UserInfo.objects.all(), owner=user) serializer = UserInfoPubkeySerializer(user_info) @@ -620,7 +620,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(data, status=status.HTTP_201_CREATED) def get_login_user(self, username): - kwargs = {User.USERNAME_FIELD: username} + kwargs = {User.USERNAME_FIELD: username.lower()} return get_object_or_404(self.get_queryset(), **kwargs) def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): From af86d877f2a26554748722b2dfa197f63020d893 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 15:40:14 +0300 Subject: [PATCH 245/511] Signup: use the shorthand version of setting an unusable password. It wasn't actually saving the unusable password before. --- django_etebase/serializers.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 94ab3e7..a77c037 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -397,13 +397,11 @@ class AuthenticationSignupSerializer(serializers.Serializer): instance = User.objects.get_by_natural_key(user_data['username']) except User.DoesNotExist: # Create the user and save the casing the user chose as the first name - instance = User.objects.create_user(**user_data, first_name=user_data['username']) + instance = User.objects.create_user(**user_data, password=None, first_name=user_data['username']) if hasattr(instance, 'userinfo'): raise serializers.ValidationError('User already exists') - instance.set_unusable_password() - try: instance.clean_fields() except django_exceptions.ValidationError as e: From 46b4f08afa6b6fe96f3fdc3aeb98e96290f3ec05 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 16:03:34 +0300 Subject: [PATCH 246/511] Signup: use the get_user_queryset function when checking if user exists. --- django_etebase/serializers.py | 4 +++- django_etebase/views.py | 18 ++++++++++++++++-- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index a77c037..29e1d4f 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -394,7 +394,9 @@ class AuthenticationSignupSerializer(serializers.Serializer): with transaction.atomic(): try: - instance = User.objects.get_by_natural_key(user_data['username']) + view = self.context.get('view', None) + user_queryset = get_user_queryset(User.objects.all(), view) + instance = user_queryset.get(**{User.USERNAME_FIELD: user_data['username'].lower()}) except User.DoesNotExist: # Create the user and save the casing the user chose as the first name instance = User.objects.create_user(**user_data, password=None, first_name=user_data['username']) diff --git a/django_etebase/views.py b/django_etebase/views.py index 327bc08..8a6ff85 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -601,6 +601,13 @@ class AuthenticationViewSet(viewsets.ViewSet): def get_queryset(self): return get_user_queryset(User.objects.all(), self) + def get_serializer_context(self): + return { + 'request': self.request, + 'format': self.format_kwarg, + 'view': self + } + def login_response_data(self, user): return { 'token': AuthToken.objects.create(user=user).key, @@ -612,7 +619,7 @@ class AuthenticationViewSet(viewsets.ViewSet): @action_decorator(detail=False, methods=['POST']) def signup(self, request, *args, **kwargs): - serializer = AuthenticationSignupSerializer(data=request.data) + serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) serializer.is_valid(raise_exception=True) user = serializer.save() @@ -748,6 +755,13 @@ class TestAuthenticationViewSet(viewsets.ViewSet): renderer_classes = BaseViewSet.renderer_classes parser_classes = BaseViewSet.parser_classes + def get_serializer_context(self): + return { + 'request': self.request, + 'format': self.format_kwarg, + 'view': self + } + def list(self, request, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) @@ -768,7 +782,7 @@ class TestAuthenticationViewSet(viewsets.ViewSet): if hasattr(user, 'userinfo'): user.userinfo.delete() - serializer = AuthenticationSignupSerializer(data=request.data) + serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) serializer.is_valid(raise_exception=True) serializer.save() From e41f8455f2b57894dde22062e52da241020639e9 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 16:08:46 +0300 Subject: [PATCH 247/511] app settings: rename the get user queryset func setting name. --- django_etebase/app_settings.py | 4 ++-- django_etebase/utils.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index 2b9da4a..b608717 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -48,8 +48,8 @@ class AppSettings: return ret @cached_property - def GET_USER_QUERYSET(self): # pylint: disable=invalid-name - get_user_queryset = self._setting("GET_USER_QUERYSET", None) + def GET_USER_QUERYSET_FUNC(self): # pylint: disable=invalid-name + get_user_queryset = self._setting("GET_USER_QUERYSET_FUNC", None) if get_user_queryset is not None: return self.import_from_str(get_user_queryset) return None diff --git a/django_etebase/utils.py b/django_etebase/utils.py index 315b82f..bce2877 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -6,7 +6,7 @@ User = get_user_model() def get_user_queryset(queryset, view): - custom_func = app_settings.GET_USER_QUERYSET + custom_func = app_settings.GET_USER_QUERYSET_FUNC if custom_func is not None: return custom_func(queryset, view) return queryset From c9463cadbad200e5411627656b8ec3f61bc57caa Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 16:20:46 +0300 Subject: [PATCH 248/511] Add support for a custom user creation function. --- django_etebase/app_settings.py | 7 +++++++ django_etebase/serializers.py | 7 +++++-- django_etebase/utils.py | 8 ++++++++ 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index b608717..3c659c8 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -54,6 +54,13 @@ class AppSettings: return self.import_from_str(get_user_queryset) return None + @cached_property + def CREATE_USER_FUNC(self): # pylint: disable=invalid-name + func = self._setting("CREATE_USER_FUNC", None) + if func is not None: + return self.import_from_str(func) + return None + @cached_property def CHALLENGE_VALID_SECONDS(self): # pylint: disable=invalid-name return self._setting("CHALLENGE_VALID_SECONDS", 60) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 29e1d4f..b3f99e0 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -20,7 +20,7 @@ from django.contrib.auth import get_user_model from django.db import transaction from rest_framework import serializers from . import models -from .utils import get_user_queryset +from .utils import get_user_queryset, create_user User = get_user_model() @@ -399,7 +399,10 @@ class AuthenticationSignupSerializer(serializers.Serializer): instance = user_queryset.get(**{User.USERNAME_FIELD: user_data['username'].lower()}) except User.DoesNotExist: # Create the user and save the casing the user chose as the first name - instance = User.objects.create_user(**user_data, password=None, first_name=user_data['username']) + try: + instance = create_user(**user_data, password=None, first_name=user_data['username'], view=view) + except Exception as e: + raise serializers.ValidationError(e) if hasattr(instance, 'userinfo'): raise serializers.ValidationError('User already exists') diff --git a/django_etebase/utils.py b/django_etebase/utils.py index bce2877..08f81ae 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -10,3 +10,11 @@ def get_user_queryset(queryset, view): if custom_func is not None: return custom_func(queryset, view) return queryset + + +def create_user(*args, **kwargs): + custom_func = app_settings.CREATE_USER_FUNC + if custom_func is not None: + return custom_func(*args, **kwargs) + _ = kwargs.pop('view') + return User.objects.create_user(*args, **kwargs) From a7268443caf1da5ed7cf627904baa6dbfac99991 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 13 Jul 2020 17:08:36 +0300 Subject: [PATCH 249/511] Add support for a modifying the chunk storage location --- django_etebase/app_settings.py | 7 +++++++ django_etebase/models.py | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index 3c659c8..33dc65f 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -61,6 +61,13 @@ class AppSettings: return self.import_from_str(func) return None + @cached_property + def CHUNK_PATH_FUNC(self): # pylint: disable=invalid-name + func = self._setting("CHUNK_PATH_FUNC", None) + if func is not None: + return self.import_from_str(func) + return None + @cached_property def CHALLENGE_VALID_SECONDS(self): # pylint: disable=invalid-name return self._setting("CHALLENGE_VALID_SECONDS", 60) diff --git a/django_etebase/models.py b/django_etebase/models.py index 397600e..f3704a3 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -21,6 +21,8 @@ from django.db.models import Q from django.utils.functional import cached_property from django.utils.crypto import get_random_string +from . import app_settings + UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{20,}$', message='Not a valid UID') @@ -79,6 +81,10 @@ class CollectionItem(models.Model): def chunk_directory_path(instance, filename): + custom_func = app_settings.CHUNK_PATH_FUNC + if custom_func is not None: + return custom_func(instance, filename) + item = instance.item col = item.collection user_id = col.owner.id From 3d6ba634ce6e4b291ae138345a81625e9759aaf6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 16 Jul 2020 10:40:30 +0300 Subject: [PATCH 250/511] Disallow + in usernames. --- myauth/models.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/myauth/models.py b/myauth/models.py index 4046b2f..611555b 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -7,10 +7,10 @@ from django.utils.translation import gettext_lazy as _ @deconstructible class UnicodeUsernameValidator(validators.RegexValidator): - regex = r'^[\w.+-]+\Z' + regex = r'^[\w.-]+\Z' message = _( 'Enter a valid username. This value may contain only letters, ' - 'numbers, and ./+/-/_ characters.' + 'numbers, and ./-/_ characters.' ) flags = 0 @@ -29,7 +29,7 @@ class User(AbstractUser): _('username'), max_length=150, unique=True, - help_text=_('Required. 150 characters or fewer. Letters, digits and ./+/-/_ only.'), + help_text=_('Required. 150 characters or fewer. Letters, digits and ./-/_ only.'), validators=[username_validator], error_messages={ 'unique': _("A user with that username already exists."), From 9c129e573187871130597b372ec045023ab3c034 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 22 Jul 2020 11:31:08 +0300 Subject: [PATCH 251/511] Collection erializer: make the item a child instead of trying to merge them. --- django_etebase/serializers.py | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index b3f99e0..bde8095 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -208,15 +208,11 @@ class CollectionSerializer(serializers.ModelSerializer): accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) - uid = serializers.CharField(source='main_item.uid') - encryptionKey = BinaryBase64Field(source='main_item.encryptionKey', required=False, default=None, allow_null=True) - etag = serializers.CharField(allow_null=True, write_only=True) - version = serializers.IntegerField(min_value=0, source='main_item.version') - content = CollectionItemRevisionSerializer(many=False, source='main_item.content') + item = CollectionItemSerializer(many=False, source='main_item') class Meta: model = models.Collection - fields = ('uid', 'version', 'accessLevel', 'encryptionKey', 'collectionKey', 'content', 'stoken', 'etag') + fields = ('item', 'accessLevel', 'collectionKey', 'stoken') def get_access_level_from_context(self, obj): request = self.context.get('request', None) @@ -228,13 +224,9 @@ class CollectionSerializer(serializers.ModelSerializer): """Function that's called when this serializer creates an item""" collection_key = validated_data.pop('collectionKey') - etag = validated_data.pop('etag') - main_item_data = validated_data.pop('main_item') - uid = main_item_data.pop('uid') - version = main_item_data.pop('version') + etag = main_item_data.pop('etag') revision_data = main_item_data.pop('content') - encryption_key = main_item_data.pop('encryptionKey') instance = self.__class__.Meta.model(**validated_data) @@ -243,8 +235,7 @@ class CollectionSerializer(serializers.ModelSerializer): raise serializers.ValidationError('etag is not None') instance.save() - main_item = models.CollectionItem.objects.create( - uid=uid, encryptionKey=encryption_key, version=version, collection=instance) + main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) instance.main_item = main_item instance.save() From 04231ebfe53c82a5715c1650530ca58d2bf7e9c3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 25 Jul 2020 09:30:40 +0300 Subject: [PATCH 252/511] Views: fix issue with iterators sometimes returning the wrong type. --- django_etebase/views.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 8a6ff85..9a71eee 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -306,11 +306,11 @@ class CollectionItemViewSet(BaseViewSet): serializer = CollectionItemRevisionSerializer(result, context=self.get_serializer_context(), many=True) - last_item = len(result) > 0 and serializer.data[-1] + iterator = serializer.data[-1]['uid'] if len(result) > 0 else None ret = { 'data': serializer.data, - 'iterator': last_item and last_item['uid'], + 'iterator': iterator, 'done': done, } return Response(ret) @@ -337,7 +337,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) new_stoken = self.get_queryset_stoken(queryset) - stoken = stoken_rev and stoken_rev.uid + stoken = getattr(stoken_rev, 'uid', None) if stoken_rev is not None else None new_stoken = new_stoken or stoken serializer = self.get_serializer(queryset, many=True) @@ -519,11 +519,11 @@ class InvitationBaseViewSet(BaseViewSet): serializer = self.get_serializer(result, many=True) - last_item = len(result) > 0 and serializer.data[-1] + iterator = serializer.data[-1]['uid'] if len(result) > 0 else None ret = { 'data': serializer.data, - 'iterator': last_item and last_item['uid'], + 'iterator': iterator, 'done': done, } From c0575cb64c64d19fd8336113bad0517912bfc81f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 30 Jul 2020 10:13:24 +0300 Subject: [PATCH 253/511] Exceptions: have correct code/status_code for every error. --- django_etebase/exceptions.py | 10 ++++++++++ django_etebase/serializers.py | 22 ++++++++++++---------- django_etebase/views.py | 15 +++++++++++---- 3 files changed, 33 insertions(+), 14 deletions(-) create mode 100644 django_etebase/exceptions.py diff --git a/django_etebase/exceptions.py b/django_etebase/exceptions.py new file mode 100644 index 0000000..d05c4e5 --- /dev/null +++ b/django_etebase/exceptions.py @@ -0,0 +1,10 @@ +from rest_framework import serializers, status + + +class EtebaseValidationError(serializers.ValidationError): + def __init__(self, code, detail, status_code=status.HTTP_400_BAD_REQUEST): + super().__init__({ + 'code': code, + 'detail': detail, + }) + self.status_code = status_code diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index bde8095..c8e93cb 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -18,10 +18,12 @@ from django.core.files.base import ContentFile from django.core import exceptions as django_exceptions from django.contrib.auth import get_user_model from django.db import transaction -from rest_framework import serializers +from rest_framework import serializers, status from . import models from .utils import get_user_queryset, create_user +from .exceptions import EtebaseValidationError + User = get_user_model() @@ -40,7 +42,7 @@ def process_revisions_for_item(item, revision_data): chunk_obj.save() else: if chunk_obj is None: - raise serializers.ValidationError('Tried to create a new chunk without content') + raise EtebaseValidationError('chunk_no_content', 'Tried to create a new chunk without content') chunks_objs.append(chunk_obj) @@ -115,7 +117,7 @@ class ChunksField(serializers.RelatedField): def to_internal_value(self, data): if data[0] is None or data[1] is None: - raise serializers.ValidationError('null is not allowed') + raise EtebaseValidationError('null is not allowed') return (data[0], b64decode_or_bytes(data[1])) @@ -161,7 +163,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): cur_etag = instance.etag if not created else None if validate_etag and cur_etag != etag: - raise serializers.ValidationError('Wrong etag. Expected {} got {}'.format(cur_etag, etag)) + raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(cur_etag, etag), status_code=status.HTTP_409_CONFLICT) if not created: # We don't have to use select_for_update here because the unique constraint on current guards against @@ -190,7 +192,7 @@ class CollectionItemDepSerializer(serializers.ModelSerializer): item = self.__class__.Meta.model.objects.get(uid=data['uid']) etag = data['etag'] if item.etag != etag: - raise serializers.ValidationError('Wrong etag. Expected {} got {}'.format(item.etag, etag)) + raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(item.etag, etag), status_code=status.HTTP_409_CONFLICT) return data @@ -232,7 +234,7 @@ class CollectionSerializer(serializers.ModelSerializer): with transaction.atomic(): if etag is not None: - raise serializers.ValidationError('etag is not None') + raise EtebaseValidationError('bad_etag', 'etag is not null') instance.save() main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) @@ -297,7 +299,7 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): request = self.context['request'] if request.user.username == value.lower(): - raise serializers.ValidationError('Inviting yourself is not allowed') + raise EtebaseValidationError('no_self_invite', 'Inviting yourself is not allowed') return value def create(self, validated_data): @@ -393,15 +395,15 @@ class AuthenticationSignupSerializer(serializers.Serializer): try: instance = create_user(**user_data, password=None, first_name=user_data['username'], view=view) except Exception as e: - raise serializers.ValidationError(e) + raise EtebaseValidationError('generic', str(e)) if hasattr(instance, 'userinfo'): - raise serializers.ValidationError('User already exists') + raise EtebaseValidationError('user_exists', 'User already exists', status_code=status.HTTP_409_CONFLICT) try: instance.clean_fields() except django_exceptions.ValidationError as e: - raise serializers.ValidationError(e) + raise EtebaseValidationError('generic', str(e)) # FIXME: send email verification models.UserInfo.objects.create(**validated_data, owner=instance) diff --git a/django_etebase/views.py b/django_etebase/views.py index 9a71eee..eb04e4b 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -72,7 +72,7 @@ from .serializers import ( UserSerializer, ) from .utils import get_user_queryset - +from .exceptions import EtebaseValidationError User = get_user_model() @@ -111,7 +111,14 @@ class BaseViewSet(viewsets.ModelViewSet): stoken = self.get_stoken_obj_id(request) if stoken is not None: - return get_object_or_404(Stoken.objects.all(), uid=stoken) + try: + return Stoken.objects.get(uid=stoken) + except Stoken.DoesNotExist: + raise EtebaseValidationError({ + 'code': 'bad_stoken', + 'detail': 'Invalid stoken.', + }, + status_code=status.HTTP_400_BAD_REQUEST) return None @@ -363,7 +370,7 @@ class CollectionItemViewSet(BaseViewSet): if stoken is not None and stoken != collection_object.stoken: content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} - return Response(content, status=status.HTTP_400_BAD_REQUEST) + return Response(content, status=status.HTTP_409_CONFLICT) items = request.data.get('items') deps = request.data.get('deps', None) @@ -387,7 +394,7 @@ class CollectionItemViewSet(BaseViewSet): "items": serializer.errors, "deps": deps_serializer.errors if deps is not None else [], }, - status=status.HTTP_400_BAD_REQUEST) + status=status.HTTP_409_CONFLICT) class CollectionItemChunkViewSet(viewsets.ViewSet): From f6af96ace6472804f52e66acafb411fa968da41b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 30 Jul 2020 10:17:26 +0300 Subject: [PATCH 254/511] Permissions: workaround DRF bug and expose exception code. --- django_etebase/permissions.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/django_etebase/permissions.py b/django_etebase/permissions.py index 6a36afb..c624404 100644 --- a/django_etebase/permissions.py +++ b/django_etebase/permissions.py @@ -25,8 +25,10 @@ class IsCollectionAdmin(permissions.BasePermission): """ Custom permission to only allow owners of a collection to view it """ - message = 'Only collection admins can perform this operation.' - code = 'admin_access_required' + message = { + 'detail': 'Only collection admins can perform this operation.', + 'code': 'admin_access_required', + } def has_permission(self, request, view): collection_uid = view.kwargs['collection_uid'] @@ -42,8 +44,10 @@ class IsCollectionAdminOrReadOnly(permissions.BasePermission): """ Custom permission to only allow owners of a collection to edit it """ - message = 'Only collection admins can edit collections.' - code = 'admin_access_required' + message = { + 'detail': 'Only collection admins can edit collections.', + 'code': 'admin_access_required', + } def has_permission(self, request, view): collection_uid = view.kwargs.get('collection_uid', None) @@ -67,8 +71,10 @@ class HasWriteAccessOrReadOnly(permissions.BasePermission): """ Custom permission to restrict write """ - message = 'You need write access to write to this collection' - code = 'no_write_access' + message = { + 'detail': 'You need write access to write to this collection', + 'code': 'no_write_access', + } def has_permission(self, request, view): collection_uid = view.kwargs['collection_uid'] From 11001ed62ce0f7ede28514f5a6efb609460fc9f6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 4 Aug 2020 13:17:48 +0300 Subject: [PATCH 255/511] Chunk serializer: fix bad error invocation. --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index c8e93cb..0e44228 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -117,7 +117,7 @@ class ChunksField(serializers.RelatedField): def to_internal_value(self, data): if data[0] is None or data[1] is None: - raise EtebaseValidationError('null is not allowed') + raise EtebaseValidationError('no_null', 'null is not allowed') return (data[0], b64decode_or_bytes(data[1])) From 1d5baece1e1244558882835dacafc1d7c8def8ac Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 4 Aug 2020 13:42:28 +0300 Subject: [PATCH 256/511] Chunk uploading: implement properly using a custom Parser. --- .../migrations/0022_auto_20200804_1059.py | 17 ++++++++++++++ django_etebase/models.py | 3 +++ django_etebase/parsers.py | 15 +++++++++++++ django_etebase/views.py | 22 ++++++++++++++----- 4 files changed, 52 insertions(+), 5 deletions(-) create mode 100644 django_etebase/migrations/0022_auto_20200804_1059.py create mode 100644 django_etebase/parsers.py diff --git a/django_etebase/migrations/0022_auto_20200804_1059.py b/django_etebase/migrations/0022_auto_20200804_1059.py new file mode 100644 index 0000000..c47e562 --- /dev/null +++ b/django_etebase/migrations/0022_auto_20200804_1059.py @@ -0,0 +1,17 @@ +# Generated by Django 3.0.3 on 2020-08-04 10:59 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0021_auto_20200626_0913'), + ] + + operations = [ + migrations.AlterUniqueTogether( + name='collectionitemchunk', + unique_together={('item', 'uid')}, + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index f3704a3..0c33301 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -100,6 +100,9 @@ class CollectionItemChunk(models.Model): def __str__(self): return self.uid + class Meta: + unique_together = ('item', 'uid') + def generate_stoken_uid(): return get_random_string(32, allowed_chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_') diff --git a/django_etebase/parsers.py b/django_etebase/parsers.py new file mode 100644 index 0000000..1ca1a70 --- /dev/null +++ b/django_etebase/parsers.py @@ -0,0 +1,15 @@ +from rest_framework.parsers import FileUploadParser + + +class ChunkUploadParser(FileUploadParser): + """ + Parser for chunk upload data. + """ + media_type = 'application/octet-stream' + + def get_filename(self, stream, media_type, parser_context): + """ + Detects the uploaded file name. + """ + view = parser_context['view'] + return parser_context['kwargs'][view.lookup_field] diff --git a/django_etebase/views.py b/django_etebase/views.py index eb04e4b..c299dc2 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -73,6 +73,7 @@ from .serializers import ( ) from .utils import get_user_queryset from .exceptions import EtebaseValidationError +from .parsers import ChunkUploadParser User = get_user_model() @@ -398,11 +399,11 @@ class CollectionItemViewSet(BaseViewSet): class CollectionItemChunkViewSet(viewsets.ViewSet): - allowed_methods = ['GET', 'POST'] + allowed_methods = ['GET', 'PUT'] authentication_classes = BaseViewSet.authentication_classes permission_classes = BaseViewSet.permission_classes renderer_classes = BaseViewSet.renderer_classes - parser_classes = (MultiPartParser, ) + parser_classes = (ChunkUploadParser, ) serializer_class = CollectionItemChunkSerializer lookup_field = 'uid' @@ -413,13 +414,24 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): user = self.request.user return queryset.filter(members__user=user) - def create(self, request, collection_uid=None, collection_item_uid=None, *args, **kwargs): + def update(self, request, *args, collection_uid=None, collection_item_uid=None, uid=None, **kwargs): col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) col_it = get_object_or_404(col.items, uid=collection_item_uid) - serializer = self.get_serializer_class()(data=request.data) + data = { + "uid": uid, + "chunkFile": request.data["file"], + } + + serializer = self.get_serializer_class()(data=data) serializer.is_valid(raise_exception=True) - serializer.save(item=col_it) + try: + serializer.save(item=col_it) + except IntegrityError: + return Response( + {"code": "chunk_exists", "detail": "Chunk already exists."}, + status=status.HTTP_409_CONFLICT + ) return Response({}, status=status.HTTP_201_CREATED) From 393b85d3ca23513a5ea7a61fdeab94f33c7eb490 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 4 Aug 2020 15:19:45 +0300 Subject: [PATCH 257/511] Chunks: move to reside under the collection. --- .../0023_collectionitemchunk_collection.py | 19 +++++++++++++ .../migrations/0024_auto_20200804_1209.py | 22 +++++++++++++++ .../migrations/0025_auto_20200804_1216.py | 27 +++++++++++++++++++ django_etebase/models.py | 9 +++---- django_etebase/serializers.py | 2 +- django_etebase/views.py | 8 +++--- 6 files changed, 77 insertions(+), 10 deletions(-) create mode 100644 django_etebase/migrations/0023_collectionitemchunk_collection.py create mode 100644 django_etebase/migrations/0024_auto_20200804_1209.py create mode 100644 django_etebase/migrations/0025_auto_20200804_1216.py diff --git a/django_etebase/migrations/0023_collectionitemchunk_collection.py b/django_etebase/migrations/0023_collectionitemchunk_collection.py new file mode 100644 index 0000000..b5d6841 --- /dev/null +++ b/django_etebase/migrations/0023_collectionitemchunk_collection.py @@ -0,0 +1,19 @@ +# Generated by Django 3.0.3 on 2020-08-04 12:08 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0022_auto_20200804_1059'), + ] + + operations = [ + migrations.AddField( + model_name='collectionitemchunk', + name='collection', + field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etebase.Collection'), + ), + ] diff --git a/django_etebase/migrations/0024_auto_20200804_1209.py b/django_etebase/migrations/0024_auto_20200804_1209.py new file mode 100644 index 0000000..54c80a3 --- /dev/null +++ b/django_etebase/migrations/0024_auto_20200804_1209.py @@ -0,0 +1,22 @@ +# Generated by Django 3.0.3 on 2020-08-04 12:09 + +from django.db import migrations + + +def change_chunk_to_collections(apps, schema_editor): + CollectionItemChunk = apps.get_model('django_etebase', 'CollectionItemChunk') + + for chunk in CollectionItemChunk.objects.all(): + chunk.collection = chunk.item.collection + chunk.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0023_collectionitemchunk_collection'), + ] + + operations = [ + migrations.RunPython(change_chunk_to_collections), + ] diff --git a/django_etebase/migrations/0025_auto_20200804_1216.py b/django_etebase/migrations/0025_auto_20200804_1216.py new file mode 100644 index 0000000..8849f53 --- /dev/null +++ b/django_etebase/migrations/0025_auto_20200804_1216.py @@ -0,0 +1,27 @@ +# Generated by Django 3.0.3 on 2020-08-04 12:16 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0024_auto_20200804_1209'), + ] + + operations = [ + migrations.AlterField( + model_name='collectionitemchunk', + name='collection', + field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etebase.Collection'), + ), + migrations.AlterUniqueTogether( + name='collectionitemchunk', + unique_together={('collection', 'uid')}, + ), + migrations.RemoveField( + model_name='collectionitemchunk', + name='item', + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 0c33301..403d2b7 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -85,23 +85,22 @@ def chunk_directory_path(instance, filename): if custom_func is not None: return custom_func(instance, filename) - item = instance.item - col = item.collection + col = instance.collection user_id = col.owner.id - return Path('user_{}'.format(user_id), col.uid, item.uid, instance.uid) + return Path('user_{}'.format(user_id), col.uid, instance.uid) class CollectionItemChunk(models.Model): uid = models.CharField(db_index=True, blank=False, null=False, max_length=60, validators=[UidValidator]) - item = models.ForeignKey(CollectionItem, related_name='chunks', on_delete=models.CASCADE) + collection = models.ForeignKey(Collection, related_name='chunks', on_delete=models.CASCADE) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) def __str__(self): return self.uid class Meta: - unique_together = ('item', 'uid') + unique_together = ('collection', 'uid') def generate_stoken_uid(): diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 0e44228..8cfddad 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -37,7 +37,7 @@ def process_revisions_for_item(item, revision_data): content = chunk[1] # If the chunk already exists we assume it's fine. Otherwise, we upload it. if chunk_obj is None: - chunk_obj = models.CollectionItemChunk(uid=uid, item=item) + chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) chunk_obj.chunkFile.save('IGNORED', ContentFile(content)) chunk_obj.save() else: diff --git a/django_etebase/views.py b/django_etebase/views.py index c299dc2..2678451 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -416,7 +416,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): def update(self, request, *args, collection_uid=None, collection_item_uid=None, uid=None, **kwargs): col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) - col_it = get_object_or_404(col.items, uid=collection_item_uid) + # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) data = { "uid": uid, @@ -426,7 +426,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer = self.get_serializer_class()(data=data) serializer.is_valid(raise_exception=True) try: - serializer.save(item=col_it) + serializer.save(collection=col) except IntegrityError: return Response( {"code": "chunk_exists", "detail": "Chunk already exists."}, @@ -441,8 +441,8 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): from django.views.static import serve col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) - col_it = get_object_or_404(col.items, uid=collection_item_uid) - chunk = get_object_or_404(col_it.chunks, uid=uid) + # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) + chunk = get_object_or_404(col.chunks, uid=uid) filename = chunk.chunkFile.path dirname = os.path.dirname(filename) From e385aa8f20ef6f160e43a5461b9603c870b5716e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 4 Aug 2020 15:37:07 +0300 Subject: [PATCH 258/511] Chunks: use a prefix of the chunk for a subdirectory. Filesystems don't handle massive directories too well, so better to split. Using the prefix of the chunk gives us a maximum of 64 * 64 = 4096 entries in the main directory. --- django_etebase/models.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index 403d2b7..7570bae 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -87,7 +87,9 @@ def chunk_directory_path(instance, filename): col = instance.collection user_id = col.owner.id - return Path('user_{}'.format(user_id), col.uid, instance.uid) + uid_prefix = instance.uid[:2] + uid_rest = instance.uid[2:] + return Path('user_{}'.format(user_id), col.uid, uid_prefix, uid_rest) class CollectionItemChunk(models.Model): From a613a326283e709a05c681f6c691c3994e91ab67 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 4 Aug 2020 15:59:31 +0300 Subject: [PATCH 259/511] prefetch: fix handling of the prefetch param. --- django_etebase/views.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 2678451..95031e3 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -190,7 +190,7 @@ class CollectionViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefetch = self.request.query_params.get('prefetch', True) + prefetch = self.request.query_params.get('prefetch', 'true') != 'false' context.update({'request': self.request, 'prefetch': prefetch}) return context @@ -256,7 +256,7 @@ class CollectionItemViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefetch = self.request.query_params.get('prefetch', True) + prefetch = self.request.query_params.get('prefetch', 'true') != 'false' context.update({'request': self.request, 'prefetch': prefetch}) return context From cf9b6f5904c22c7a03e719d914b308da22261371 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 4 Aug 2020 17:44:57 +0300 Subject: [PATCH 260/511] Prefetch: change the type of value prefetch accept. It's 'auto' by default, but can be changed to 'medium' and soon another value. --- django_etebase/serializers.py | 2 +- django_etebase/views.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 8cfddad..40f6068 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -109,7 +109,7 @@ class UserSlugRelatedField(serializers.SlugRelatedField): class ChunksField(serializers.RelatedField): def to_representation(self, obj): obj = obj.chunk - if self.context.get('prefetch'): + if self.context.get('prefetch') == 'auto': with open(obj.chunkFile.path, 'rb') as f: return (obj.uid, f.read()) else: diff --git a/django_etebase/views.py b/django_etebase/views.py index 95031e3..9d76d08 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -190,7 +190,7 @@ class CollectionViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefetch = self.request.query_params.get('prefetch', 'true') != 'false' + prefetch = self.request.query_params.get('prefetch', 'auto') context.update({'request': self.request, 'prefetch': prefetch}) return context @@ -256,7 +256,7 @@ class CollectionItemViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefetch = self.request.query_params.get('prefetch', 'true') != 'false' + prefetch = self.request.query_params.get('prefetch', 'auto') context.update({'request': self.request, 'prefetch': prefetch}) return context From 5af2aeda7e44e1b100b51df6da60e9ee7b2b2dcf Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 18 Aug 2020 12:02:56 +0300 Subject: [PATCH 261/511] Add an endpoint to know if a server is an etebase server or not. Very useful for when migrating people from legacy EteSync apps because we can automatically know if they are running a self-hosted etesync or etebase server. --- django_etebase/views.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/django_etebase/views.py b/django_etebase/views.py index 9d76d08..1e9b8b5 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -687,6 +687,10 @@ class AuthenticationViewSet(viewsets.ViewSet): return None + @action_decorator(detail=False, methods=['GET']) + def is_etebase(self, request, *args, **kwargs): + return Response({}, status=status.HTTP_200_OK) + @action_decorator(detail=False, methods=['POST']) def login_challenge(self, request, *args, **kwargs): from datetime import datetime From 693a5ec778b9a4645dc52de65533f1e1591e752f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 18 Aug 2020 12:04:42 +0300 Subject: [PATCH 262/511] Login: return an UNAUTHORIZED (401) error on bad username/password, not 400. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 1e9b8b5..a39d443 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -683,7 +683,7 @@ class AuthenticationViewSet(viewsets.ViewSet): try: verify_key.verify(response_raw, signature) except nacl.exceptions.BadSignatureError: - return Response({'code': 'login_bad_signature'}, status=status.HTTP_400_BAD_REQUEST) + return Response({'code': 'login_bad_signature'}, status=status.HTTP_401_UNAUTHORIZED) return None From 8593ab13578dec500b5768c8e21f3d1afc649630 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 18 Aug 2020 12:24:30 +0300 Subject: [PATCH 263/511] Login: add a user visible error on password failure. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index a39d443..c45630b 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -683,7 +683,7 @@ class AuthenticationViewSet(viewsets.ViewSet): try: verify_key.verify(response_raw, signature) except nacl.exceptions.BadSignatureError: - return Response({'code': 'login_bad_signature'}, status=status.HTTP_401_UNAUTHORIZED) + return Response({'code': 'login_bad_signature', 'detail': 'Wrong password for user.'}, status=status.HTTP_401_UNAUTHORIZED) return None From 2327466113838e1594e25a787183fb0a2cea23eb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 28 Aug 2020 13:55:15 +0300 Subject: [PATCH 264/511] Invitations: error when trying to invite oneself. --- django_etebase/views.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/django_etebase/views.py b/django_etebase/views.py index c45630b..c8a98fc 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -568,6 +568,10 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): except Collection.DoesNotExist: raise Http404('Collection does not exist') + if request.user == serializer.validated_data.get('user'): + content = {'code': 'self_invite', 'detail': 'Inviting yourself is invalid'} + return Response(content, status=status.HTTP_400_BAD_REQUEST) + if not permissions.is_collection_admin(collection, request.user): raise PermissionDenied('User is not an admin of this collection') From bf22b1676f39ff05ca743fcd7f4789896997dc3f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 2 Sep 2020 11:07:43 +0300 Subject: [PATCH 265/511] Serializers: improve field serialization errors. --- django_etebase/serializers.py | 88 ++++++++++++++++++++++++++++------- 1 file changed, 72 insertions(+), 16 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 40f6068..356da82 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -121,13 +121,69 @@ class ChunksField(serializers.RelatedField): return (data[0], b64decode_or_bytes(data[1])) -class CollectionItemChunkSerializer(serializers.ModelSerializer): +class BetterErrorsMixin: + @property + def errors(self): + nice = [] + errors = super().errors + for error_type in errors: + if error_type == 'non_field_errors': + nice.extend( + self.flatten_errors(None, errors[error_type]) + ) + else: + nice.extend( + self.flatten_errors(error_type, errors[error_type]) + ) + if nice: + return {'code': 'field_errors', + 'message': 'Field validations failed.', + 'errors': nice} + return {} + + def flatten_errors(self, field_name, errors): + ret = [] + if isinstance(errors, dict): + for error_key in errors: + error = errors[error_key] + ret.extend(self.flatten_errors("{}.{}".format(field_name, error_key), error)) + else: + for error in errors: + if hasattr(error, 'detail'): + message = error.detail[0] + elif hasattr(error, 'message'): + message = error.message + else: + message = str(error) + ret.append({ + 'field': field_name, + 'code': error.code, + 'message': message, + }) + return ret + + def transform_validation_error(self, prefix, err): + if hasattr(err, 'error_dict'): + errors = self.flatten_errors(prefix, err.error_dict) + elif not hasattr(err, 'message'): + errors = self.flatten_errors(prefix, err.error_list) + else: + raise EtebaseValidationError(err.code, err.message) + + raise serializers.ValidationError({ + 'code': 'field_errors', + 'message': 'Field validations failed.', + 'errors': errors, + }) + + +class CollectionItemChunkSerializer(BetterErrorsMixin, serializers.ModelSerializer): class Meta: model = models.CollectionItemChunk fields = ('uid', 'chunkFile') -class CollectionItemRevisionSerializer(serializers.ModelSerializer): +class CollectionItemRevisionSerializer(BetterErrorsMixin, serializers.ModelSerializer): chunks = ChunksField( source='chunks_relation', queryset=models.RevisionChunkRelation.objects.all(), @@ -140,7 +196,7 @@ class CollectionItemRevisionSerializer(serializers.ModelSerializer): fields = ('chunks', 'meta', 'uid', 'deleted') -class CollectionItemSerializer(serializers.ModelSerializer): +class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): encryptionKey = BinaryBase64Field(required=False, default=None, allow_null=True) etag = serializers.CharField(allow_null=True, write_only=True) content = CollectionItemRevisionSerializer(many=False) @@ -181,7 +237,7 @@ class CollectionItemSerializer(serializers.ModelSerializer): raise NotImplementedError() -class CollectionItemDepSerializer(serializers.ModelSerializer): +class CollectionItemDepSerializer(BetterErrorsMixin, serializers.ModelSerializer): etag = serializers.CharField() class Meta: @@ -197,7 +253,7 @@ class CollectionItemDepSerializer(serializers.ModelSerializer): return data -class CollectionItemBulkGetSerializer(serializers.ModelSerializer): +class CollectionItemBulkGetSerializer(BetterErrorsMixin, serializers.ModelSerializer): etag = serializers.CharField(required=False) class Meta: @@ -205,7 +261,7 @@ class CollectionItemBulkGetSerializer(serializers.ModelSerializer): fields = ('uid', 'etag') -class CollectionSerializer(serializers.ModelSerializer): +class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): collectionKey = CollectionEncryptionKeyField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) @@ -257,7 +313,7 @@ class CollectionSerializer(serializers.ModelSerializer): raise NotImplementedError() -class CollectionMemberSerializer(serializers.ModelSerializer): +class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer): username = UserSlugRelatedField( source='user', read_only=True, @@ -282,7 +338,7 @@ class CollectionMemberSerializer(serializers.ModelSerializer): return instance -class CollectionInvitationSerializer(serializers.ModelSerializer): +class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSerializer): username = UserSlugRelatedField( source='user', queryset=User.objects @@ -320,7 +376,7 @@ class CollectionInvitationSerializer(serializers.ModelSerializer): return instance -class InvitationAcceptSerializer(serializers.Serializer): +class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): encryptionKey = BinaryBase64Field() def create(self, validated_data): @@ -348,7 +404,7 @@ class InvitationAcceptSerializer(serializers.Serializer): raise NotImplementedError() -class UserSerializer(serializers.ModelSerializer): +class UserSerializer(BetterErrorsMixin, serializers.ModelSerializer): pubkey = BinaryBase64Field(source='userinfo.pubkey') encryptedContent = BinaryBase64Field(source='userinfo.encryptedContent') @@ -357,7 +413,7 @@ class UserSerializer(serializers.ModelSerializer): fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, 'pubkey', 'encryptedContent') -class UserInfoPubkeySerializer(serializers.ModelSerializer): +class UserInfoPubkeySerializer(BetterErrorsMixin, serializers.ModelSerializer): pubkey = BinaryBase64Field() class Meta: @@ -365,7 +421,7 @@ class UserInfoPubkeySerializer(serializers.ModelSerializer): fields = ('pubkey', ) -class UserSignupSerializer(serializers.ModelSerializer): +class UserSignupSerializer(BetterErrorsMixin, serializers.ModelSerializer): class Meta: model = User fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) @@ -374,7 +430,7 @@ class UserSignupSerializer(serializers.ModelSerializer): } -class AuthenticationSignupSerializer(serializers.Serializer): +class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): user = UserSignupSerializer(many=False) salt = BinaryBase64Field() loginPubkey = BinaryBase64Field() @@ -403,7 +459,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): try: instance.clean_fields() except django_exceptions.ValidationError as e: - raise EtebaseValidationError('generic', str(e)) + self.transform_validation_error("user", e) # FIXME: send email verification models.UserInfo.objects.create(**validated_data, owner=instance) @@ -414,7 +470,7 @@ class AuthenticationSignupSerializer(serializers.Serializer): raise NotImplementedError() -class AuthenticationLoginChallengeSerializer(serializers.Serializer): +class AuthenticationLoginChallengeSerializer(BetterErrorsMixin, serializers.Serializer): username = serializers.CharField(required=True) def create(self, validated_data): @@ -424,7 +480,7 @@ class AuthenticationLoginChallengeSerializer(serializers.Serializer): raise NotImplementedError() -class AuthenticationLoginSerializer(serializers.Serializer): +class AuthenticationLoginSerializer(BetterErrorsMixin, serializers.Serializer): response = BinaryBase64Field() signature = BinaryBase64Field() From 7ab9513e055111aff68121ede0a27bb1c42c5d3c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 2 Sep 2020 11:11:17 +0300 Subject: [PATCH 266/511] Serializers: rename message to detail to conform with the rest of the API. This was a mistake in the previous commit. --- django_etebase/serializers.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 356da82..036da1e 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -137,7 +137,7 @@ class BetterErrorsMixin: ) if nice: return {'code': 'field_errors', - 'message': 'Field validations failed.', + 'detail': 'Field validations failed.', 'errors': nice} return {} @@ -158,7 +158,7 @@ class BetterErrorsMixin: ret.append({ 'field': field_name, 'code': error.code, - 'message': message, + 'detail': message, }) return ret @@ -172,7 +172,7 @@ class BetterErrorsMixin: raise serializers.ValidationError({ 'code': 'field_errors', - 'message': 'Field validations failed.', + 'detail': 'Field validations failed.', 'errors': errors, }) From 42a72ce5c7bda317a2fcf2218704540f89544c6f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 2 Sep 2020 12:50:47 +0300 Subject: [PATCH 267/511] Serializers user signup: correctly handle EtebaseValidationErrors. Don't coerce them to strings --- django_etebase/serializers.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 036da1e..babf7d0 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -450,6 +450,8 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): # Create the user and save the casing the user chose as the first name try: instance = create_user(**user_data, password=None, first_name=user_data['username'], view=view) + except EtebaseValidationError as e: + raise e except Exception as e: raise EtebaseValidationError('generic', str(e)) From 43569727f4dd0891636a01c851b8e3d5a92b0cfd Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 2 Sep 2020 12:54:27 +0300 Subject: [PATCH 268/511] Signup: send a signal on account signup. --- django_etebase/serializers.py | 1 - django_etebase/signals.py | 3 +++ django_etebase/views.py | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 django_etebase/signals.py diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index babf7d0..35b3562 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -462,7 +462,6 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): instance.clean_fields() except django_exceptions.ValidationError as e: self.transform_validation_error("user", e) - # FIXME: send email verification models.UserInfo.objects.create(**validated_data, owner=instance) diff --git a/django_etebase/signals.py b/django_etebase/signals.py new file mode 100644 index 0000000..03dbed5 --- /dev/null +++ b/django_etebase/signals.py @@ -0,0 +1,3 @@ +from django.dispatch import Signal + +user_signed_up = Signal(providing_args=['request', 'user']) diff --git a/django_etebase/views.py b/django_etebase/views.py index c8a98fc..97cb2f0 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -74,6 +74,7 @@ from .serializers import ( from .utils import get_user_queryset from .exceptions import EtebaseValidationError from .parsers import ChunkUploadParser +from .signals import user_signed_up User = get_user_model() @@ -646,6 +647,8 @@ class AuthenticationViewSet(viewsets.ViewSet): serializer.is_valid(raise_exception=True) user = serializer.save() + user_signed_up.send(sender=user.__class__, request=request, user=user) + data = self.login_response_data(user) return Response(data, status=status.HTTP_201_CREATED) From d90931fbe5b67112aba8c4976fa52c5310222f1e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 7 Sep 2020 11:02:40 +0300 Subject: [PATCH 269/511] Make access level an int instead of a string. We started with a string because we thought it could maybe provide more flexibility, though really, an int makes much more sense, especially on all the platforms etebase runs nowadays. --- .../migrations/0026_auto_20200907_0752.py | 23 +++++++++++ .../migrations/0027_auto_20200907_0752.py | 23 +++++++++++ .../migrations/0028_auto_20200907_0754.py | 39 +++++++++++++++++++ .../migrations/0029_auto_20200907_0801.py | 21 ++++++++++ django_etebase/models.py | 14 +++---- 5 files changed, 112 insertions(+), 8 deletions(-) create mode 100644 django_etebase/migrations/0026_auto_20200907_0752.py create mode 100644 django_etebase/migrations/0027_auto_20200907_0752.py create mode 100644 django_etebase/migrations/0028_auto_20200907_0754.py create mode 100644 django_etebase/migrations/0029_auto_20200907_0801.py diff --git a/django_etebase/migrations/0026_auto_20200907_0752.py b/django_etebase/migrations/0026_auto_20200907_0752.py new file mode 100644 index 0000000..38c0b92 --- /dev/null +++ b/django_etebase/migrations/0026_auto_20200907_0752.py @@ -0,0 +1,23 @@ +# Generated by Django 3.1 on 2020-09-07 07:52 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0025_auto_20200804_1216'), + ] + + operations = [ + migrations.RenameField( + model_name='collectioninvitation', + old_name='accessLevel', + new_name='accessLevelOld', + ), + migrations.RenameField( + model_name='collectionmember', + old_name='accessLevel', + new_name='accessLevelOld', + ), + ] diff --git a/django_etebase/migrations/0027_auto_20200907_0752.py b/django_etebase/migrations/0027_auto_20200907_0752.py new file mode 100644 index 0000000..d822d3d --- /dev/null +++ b/django_etebase/migrations/0027_auto_20200907_0752.py @@ -0,0 +1,23 @@ +# Generated by Django 3.1 on 2020-09-07 07:52 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0026_auto_20200907_0752'), + ] + + operations = [ + migrations.AddField( + model_name='collectioninvitation', + name='accessLevel', + field=models.IntegerField(choices=[(0, 'Read Only'), (1, 'Admin'), (2, 'Read Write')], default=0), + ), + migrations.AddField( + model_name='collectionmember', + name='accessLevel', + field=models.IntegerField(choices=[(0, 'Read Only'), (1, 'Admin'), (2, 'Read Write')], default=0), + ), + ] diff --git a/django_etebase/migrations/0028_auto_20200907_0754.py b/django_etebase/migrations/0028_auto_20200907_0754.py new file mode 100644 index 0000000..cb62e63 --- /dev/null +++ b/django_etebase/migrations/0028_auto_20200907_0754.py @@ -0,0 +1,39 @@ +# Generated by Django 3.1 on 2020-09-07 07:54 + +from django.db import migrations + +from django_etebase.models import AccessLevels + + +def change_access_level_to_int(apps, schema_editor): + CollectionMember = apps.get_model('django_etebase', 'CollectionMember') + CollectionInvitation = apps.get_model('django_etebase', 'CollectionInvitation') + + for member in CollectionMember.objects.all(): + if member.accessLevelOld == 'adm': + member.accessLevel = AccessLevels.ADMIN + elif member.accessLevelOld == 'rw': + member.accessLevel = AccessLevels.READ_WRITE + elif member.accessLevelOld == 'ro': + member.accessLevel = AccessLevels.READ_ONLY + member.save() + + for invitation in CollectionInvitation.objects.all(): + if invitation.accessLevelOld == 'adm': + invitation.accessLevel = AccessLevels.ADMIN + elif invitation.accessLevelOld == 'rw': + invitation.accessLevel = AccessLevels.READ_WRITE + elif invitation.accessLevelOld == 'ro': + invitation.accessLevel = AccessLevels.READ_ONLY + invitation.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0027_auto_20200907_0752'), + ] + + operations = [ + migrations.RunPython(change_access_level_to_int), + ] diff --git a/django_etebase/migrations/0029_auto_20200907_0801.py b/django_etebase/migrations/0029_auto_20200907_0801.py new file mode 100644 index 0000000..7cd54d4 --- /dev/null +++ b/django_etebase/migrations/0029_auto_20200907_0801.py @@ -0,0 +1,21 @@ +# Generated by Django 3.1 on 2020-09-07 08:01 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0028_auto_20200907_0754'), + ] + + operations = [ + migrations.RemoveField( + model_name='collectioninvitation', + name='accessLevelOld', + ), + migrations.RemoveField( + model_name='collectionmember', + name='accessLevelOld', + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 7570bae..797aa0f 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -138,10 +138,10 @@ class RevisionChunkRelation(models.Model): ordering = ('id', ) -class AccessLevels(models.TextChoices): - ADMIN = 'adm' - READ_WRITE = 'rw' - READ_ONLY = 'ro' +class AccessLevels(models.IntegerChoices): + READ_ONLY = 0 + ADMIN = 1 + READ_WRITE = 2 class CollectionMember(models.Model): @@ -149,8 +149,7 @@ class CollectionMember(models.Model): collection = models.ForeignKey(Collection, related_name='members', on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) - accessLevel = models.CharField( - max_length=3, + accessLevel = models.IntegerField( choices=AccessLevels.choices, default=AccessLevels.READ_ONLY, ) @@ -195,8 +194,7 @@ class CollectionInvitation(models.Model): user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name='incoming_invitations', on_delete=models.CASCADE) signedEncryptionKey = models.BinaryField(editable=False, blank=False, null=False) - accessLevel = models.CharField( - max_length=3, + accessLevel = models.IntegerField( choices=AccessLevels.choices, default=AccessLevels.READ_ONLY, ) From a85e8168101441dfd535c3391bc893527f43f4ac Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 9 Sep 2020 17:07:32 +0300 Subject: [PATCH 270/511] User not found: return a 401 instead of a 404. --- django_etebase/views.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 97cb2f0..7a6ea7d 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -30,6 +30,7 @@ from rest_framework.decorators import action as action_decorator from rest_framework.response import Response from rest_framework.parsers import JSONParser, FormParser, MultiPartParser from rest_framework.renderers import BrowsableAPIRenderer +from rest_framework.exceptions import AuthenticationFailed import nacl.encoding import nacl.signing @@ -654,7 +655,11 @@ class AuthenticationViewSet(viewsets.ViewSet): def get_login_user(self, username): kwargs = {User.USERNAME_FIELD: username.lower()} - return get_object_or_404(self.get_queryset(), **kwargs) + try: + return self.get_queryset().get(**kwargs) + except User.DoesNotExist: + raise AuthenticationFailed({'code': 'user_not_found', 'detail': 'User not found'}) + def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): from datetime import datetime From 9c6a7e94282d29be9129eb85f99bebe4fa3512d2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 10 Sep 2020 13:31:54 +0300 Subject: [PATCH 271/511] Login: fix server error when trying to login to users without userinfo. --- django_etebase/views.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 7a6ea7d..636287e 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -656,7 +656,10 @@ class AuthenticationViewSet(viewsets.ViewSet): def get_login_user(self, username): kwargs = {User.USERNAME_FIELD: username.lower()} try: - return self.get_queryset().get(**kwargs) + user = self.get_queryset().get(**kwargs) + if not hasattr(user, 'userinfo'): + raise AuthenticationFailed({'code': 'user_not_init', 'detail': 'User not properly init'}) + return user except User.DoesNotExist: raise AuthenticationFailed({'code': 'user_not_found', 'detail': 'User not found'}) From 5785f803ac6bf676783903bc0478a99ceb3e80d4 Mon Sep 17 00:00:00 2001 From: Pierre-Alain TORET Date: Thu, 10 Sep 2020 18:51:25 +0300 Subject: [PATCH 272/511] Port over easyconfig from the etesync server code. Migrated by Tom, but kept the credit to daftaupe --- etebase-server.ini.example | 17 +++++++++++ etebase_server/settings.py | 62 +++++++++++++++++++++++++++++--------- etebase_server/utils.py | 25 +++++++++++++++ 3 files changed, 89 insertions(+), 15 deletions(-) create mode 100644 etebase-server.ini.example create mode 100644 etebase_server/utils.py diff --git a/etebase-server.ini.example b/etebase-server.ini.example new file mode 100644 index 0000000..2b4682a --- /dev/null +++ b/etebase-server.ini.example @@ -0,0 +1,17 @@ +[global] +secret_file = secret.txt +debug = false +;Advanced options, only uncomment if you know what you're doing: +;static_root = /path/to/static +;static_url = /static/ +;media_root = /path/to/media +;media_url = /user-media/ +;language_code = en-us +;time_zone = UTC + +[allowed_hosts] +allowed_host1 = example.com + +[database] +engine = django.db.backends.sqlite3 +name = db.sqlite3 diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 94ca4d2..d5853a0 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -11,6 +11,8 @@ https://docs.djangoproject.com/en/3.0/ref/settings/ """ import os +import configparser +from .utils import get_secret_from_file # Build paths inside the project like this: os.path.join(BASE_DIR, ...) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -21,14 +23,27 @@ AUTH_USER_MODEL = 'myauth.User' # Quick-start development settings - unsuitable for production # See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/ -# Should be set in the site specific settings -# SECRET_KEY = +# SECURITY WARNING: keep the secret key used in production secret! +# See secret.py for how this is generated; uses a file 'secret.txt' in the root +# directory +SECRET_FILE = os.path.join(BASE_DIR, "secret.txt") # SECURITY WARNING: don't run with debug turned on in production! DEBUG = False ALLOWED_HOSTS = [] +# Database +# https://docs.djangoproject.com/en/2.0/ref/settings/#databases + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.environ.get('ETEBASE_DB_PATH', + os.path.join(BASE_DIR, 'db.sqlite3')), + } +} + # Application definition @@ -81,18 +96,6 @@ TEMPLATES = [ WSGI_APPLICATION = 'etebase_server.wsgi.application' -# Database -# https://docs.djangoproject.com/en/3.0/ref/settings/#databases - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': os.environ.get('ETEBASE_DB_PATH', - os.path.join(BASE_DIR, 'db.sqlite3')), - } -} - - # Password validation # https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators @@ -131,17 +134,46 @@ CORS_ORIGIN_ALLOW_ALL = True # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/3.0/howto/static-files/ -STATIC_ROOT = os.environ.get('DJANGO_STATIC_ROOT', os.path.join(BASE_DIR, 'assets')) STATIC_URL = '/static/' +STATIC_ROOT = os.environ.get('DJANGO_STATIC_ROOT', os.path.join(BASE_DIR, 'static')) MEDIA_ROOT = os.environ.get('DJANGO_MEDIA_ROOT', os.path.join(BASE_DIR, 'media')) MEDIA_URL = '/user-media/' + +# Define where to find configuration files +config_locations = ['etebase-server.ini', '/etc/etebase-server/etebase-server.ini'] +# Use config file if present +if any(os.path.isfile(x) for x in config_locations): + config = configparser.ConfigParser() + config.read(config_locations) + + section = config['global'] + + SECRET_FILE = section.get('secret_file', SECRET_FILE) + STATIC_ROOT = section.get('static_root', STATIC_ROOT) + STATIC_URL = section.get('static_url', STATIC_URL) + MEDIA_ROOT = section.get('media_root', MEDIA_ROOT) + MEDIA_URL = section.get('media_url', MEDIA_URL) + LANGUAGE_CODE = section.get('language_code', LANGUAGE_CODE) + TIME_ZONE = section.get('time_zone', TIME_ZONE) + DEBUG = section.getboolean('debug', DEBUG) + + if 'allowed_hosts' in config: + ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')] + + if 'database' in config: + DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } } + ETEBASE_API_PERMISSIONS = ('rest_framework.permissions.IsAuthenticated', ) ETEBASE_API_AUTHENTICATORS = ('django_etebase.token_auth.authentication.TokenAuthentication', 'rest_framework.authentication.SessionAuthentication') +# Make an `etebase_server_settings` module available to override settings. try: from etebase_server_settings import * except ImportError: pass + +if 'SECRET_KEY' not in locals(): + SECRET_KEY = get_secret_from_file(SECRET_FILE) diff --git a/etebase_server/utils.py b/etebase_server/utils.py new file mode 100644 index 0000000..21c99f2 --- /dev/null +++ b/etebase_server/utils.py @@ -0,0 +1,25 @@ +# Copyright © 2017 Tom Hacohen +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, version 3. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +from django.core.management import utils + +def get_secret_from_file(path): + try: + with open(path, "r") as f: + return f.read().strip() + except EnvironmentError: + with open(path, "w") as f: + secret_key = utils.get_random_secret_key() + f.write(secret_key) + return secret_key From 38e0700ac0557a8dfa14ebddf5be21da65bb1247 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 10 Sep 2020 18:54:18 +0300 Subject: [PATCH 273/511] Update django and remove unused deps. --- etebase_server/settings.py | 1 - requirements.in/base.txt | 2 -- requirements.txt | 13 +++---------- 3 files changed, 3 insertions(+), 13 deletions(-) diff --git a/etebase_server/settings.py b/etebase_server/settings.py index d5853a0..7af0c60 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -56,7 +56,6 @@ INSTALLED_APPS = [ 'django.contrib.staticfiles', 'corsheaders', 'rest_framework', - 'fullurl', 'myauth.apps.MyauthConfig', 'django_etebase.apps.DjangoEtebaseConfig', 'django_etebase.token_auth.apps.TokenAuthConfig', diff --git a/requirements.in/base.txt b/requirements.in/base.txt index d27e110..7d5bf7e 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -1,7 +1,5 @@ django -django-anymail django-cors-headers -django-fullurl djangorestframework drf-nested-routers msgpack diff --git a/requirements.txt b/requirements.txt index cd61ff1..f6c8ed4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,23 +4,16 @@ # # pip-compile --output-file=requirements.txt requirements.in/base.txt # -asgiref==3.2.3 # via django -certifi==2019.11.28 # via requests +asgiref==3.2.10 # via django cffi==1.14.0 # via pynacl -chardet==3.0.4 # via requests -django-anymail==7.0.0 # via -r requirements.in/base.txt django-cors-headers==3.2.1 # via -r requirements.in/base.txt -django-fullurl==1.0 # via -r requirements.in/base.txt -django==3.0.3 # via -r requirements.in/base.txt, django-anymail, django-cors-headers, django-fullurl, djangorestframework, drf-nested-routers +django==3.1.1 # via -r requirements.in/base.txt, django-cors-headers, djangorestframework, drf-nested-routers djangorestframework==3.11.0 # via -r requirements.in/base.txt, drf-nested-routers drf-nested-routers==0.91 # via -r requirements.in/base.txt -idna==2.8 # via requests msgpack==1.0.0 # via -r requirements.in/base.txt psycopg2-binary==2.8.4 # via -r requirements.in/base.txt pycparser==2.20 # via cffi pynacl==1.3.0 # via -r requirements.in/base.txt pytz==2019.3 # via django -requests==2.22.0 # via django-anymail -six==1.14.0 # via django-anymail, pynacl +six==1.14.0 # via pynacl sqlparse==0.3.0 # via django -urllib3==1.25.8 # via requests From b9f20d251a3be72b786844b6ba697aaa01b44158 Mon Sep 17 00:00:00 2001 From: "Prof. Jayanth R Varma" Date: Wed, 7 Nov 2018 02:18:46 +0530 Subject: [PATCH 274/511] Add example config for using using nginx with uwsgi --- example-configs/nginx-uwsgi/etesync.ini | 15 ++++++++++ .../nginx-uwsgi/my.server.name.conf | 30 +++++++++++++++++++ example-configs/nginx-uwsgi/readme.md | 20 +++++++++++++ example-configs/nginx-uwsgi/uwsgi.service | 15 ++++++++++ 4 files changed, 80 insertions(+) create mode 100644 example-configs/nginx-uwsgi/etesync.ini create mode 100644 example-configs/nginx-uwsgi/my.server.name.conf create mode 100644 example-configs/nginx-uwsgi/readme.md create mode 100644 example-configs/nginx-uwsgi/uwsgi.service diff --git a/example-configs/nginx-uwsgi/etesync.ini b/example-configs/nginx-uwsgi/etesync.ini new file mode 100644 index 0000000..e79eeee --- /dev/null +++ b/example-configs/nginx-uwsgi/etesync.ini @@ -0,0 +1,15 @@ +# uwsgi configuration file +# typical location of this file would be /etc/uwsgi/sites/etesync.ini + +[uwsgi] +socket = /path/to/etesync_server.sock +chown-socket = EtesyncUser:www-data +chmod-socket = 660 +vacuum = true + + +uid = EtesyncUser +chdir = /path/to/etesync +home = %(chdir)/.venv +module = etesync_server.wsgi +master = true diff --git a/example-configs/nginx-uwsgi/my.server.name.conf b/example-configs/nginx-uwsgi/my.server.name.conf new file mode 100644 index 0000000..b5b019d --- /dev/null +++ b/example-configs/nginx-uwsgi/my.server.name.conf @@ -0,0 +1,30 @@ +# nginx configuration for etesync server running on https://my.server.name +# typical location of this file would be /etc/nginx/sites-available/my.server.name.conf + +server { + server_name my.server.name; + + root /srv/http/etesync_server; + + client_max_body_size 5M; + + location /static { + expires 1y; + try_files $uri $uri/ =404; + } + + location / { + uwsgi_pass unix:/path/to/etesync_server.sock; + include uwsgi_params; + } + + # change 443 to say 9443 to run on a non standard port + listen 443 ssl; + listen [::]:443 ssl; + # Enable these two instead of the two above if your nginx supports http2 + # listen 443 ssl http2; + # listen [::]:443 ssl http2; + + ssl_certificate /path/to/certificate-file + ssl_certificate_key /path/to/certificate-key-file + # other ssl directives as needed diff --git a/example-configs/nginx-uwsgi/readme.md b/example-configs/nginx-uwsgi/readme.md new file mode 100644 index 0000000..dad98b6 --- /dev/null +++ b/example-configs/nginx-uwsgi/readme.md @@ -0,0 +1,20 @@ +# Running `etesync` under `nginx` and `uwsgi` + +This configuration assumes that etesync server has been installed in the home folder of a non privileged user +called `EtesyncUser` following the instructions in . Also that static +files have been collected at `/srv/http/etesync_server` by running the following commands: + + sudo mkdir -p /srv/http/etesync_server/static + sudo chown -R EtesyncUser /srv/http/etesync_server + sudo su EtesyncUser + cd /path/to/etesync + ln -s /srv/http/etesync_server/static static + ./manage.py collectstatic + +It is also assumed that `nginx` and `uwsgi` have been installed system wide by `root`, and that `nginx` is running as user/group `www-data`. + +In this setup, `uwsgi` running as a `systemd` service as `root` creates a unix socket with read-write access +to both `EtesyncUser` and `nginx`. It then drops its `root` privilege and runs `etesync` as `EtesyncUser`. + +`nginx` listens on the `https` port (or a non standard port `https` port if desired), delivers static pages directly +and for everything else, communicates with `etesync` over the unix socket. diff --git a/example-configs/nginx-uwsgi/uwsgi.service b/example-configs/nginx-uwsgi/uwsgi.service new file mode 100644 index 0000000..9941ec3 --- /dev/null +++ b/example-configs/nginx-uwsgi/uwsgi.service @@ -0,0 +1,15 @@ +# systemd unit for running uwsgi in emperor mode +# typical location of this file would be /etc/systemd/system/uwsgi.service + +[Unit] +Description=uWSGI Emperor service + +[Service] +ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/sites +Restart=always +KillSignal=SIGQUIT +Type=notify +NotifyAccess=all + +[Install] +WantedBy=multi-user.target From 9efb8d4c4090be223d3a7416fe4a9aa83052fc95 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 10 Sep 2020 19:20:52 +0300 Subject: [PATCH 275/511] Update example-configs to etebase. --- example-configs/nginx-uwsgi/README.md | 22 +++++++++++++++++++ example-configs/nginx-uwsgi/etebase.ini | 15 +++++++++++++ example-configs/nginx-uwsgi/etesync.ini | 15 ------------- .../nginx-uwsgi/my.server.name.conf | 22 ++++++++++++------- example-configs/nginx-uwsgi/readme.md | 20 ----------------- 5 files changed, 51 insertions(+), 43 deletions(-) create mode 100644 example-configs/nginx-uwsgi/README.md create mode 100644 example-configs/nginx-uwsgi/etebase.ini delete mode 100644 example-configs/nginx-uwsgi/etesync.ini delete mode 100644 example-configs/nginx-uwsgi/readme.md diff --git a/example-configs/nginx-uwsgi/README.md b/example-configs/nginx-uwsgi/README.md new file mode 100644 index 0000000..55b5fa5 --- /dev/null +++ b/example-configs/nginx-uwsgi/README.md @@ -0,0 +1,22 @@ +# Running `etebase` under `nginx` and `uwsgi` + +This configuration assumes that etebase server has been installed in the home folder of a non privileged user +called `EtebaseUser` following the instructions in . Also that static +files have been collected at `/srv/http/etebase_server` by running the following commands: + +```shell +sudo mkdir -p /srv/http/etebase_server/static +sudo chown -R EtebaseUser /srv/http/etebase_server +sudo su EtebaseUser +cd /path/to/etebase +ln -s /srv/http/etebase_server/static static +./manage.py collectstatic +``` + +It is also assumed that `nginx` and `uwsgi` have been installed system wide by `root`, and that `nginx` is running as user/group `www-data`. + +In this setup, `uwsgi` running as a `systemd` service as `root` creates a unix socket with read-write access +to both `EtebaseUser` and `nginx`. It then drops its `root` privilege and runs `etebase` as `EtebaseUser`. + +`nginx` listens on the `https` port (or a non standard port `https` port if desired), delivers static pages directly +and for everything else, communicates with `etebase` over the unix socket. diff --git a/example-configs/nginx-uwsgi/etebase.ini b/example-configs/nginx-uwsgi/etebase.ini new file mode 100644 index 0000000..a2ebe97 --- /dev/null +++ b/example-configs/nginx-uwsgi/etebase.ini @@ -0,0 +1,15 @@ +# uwsgi configuration file +# typical location of this file would be /etc/uwsgi/sites/etebase.ini + +[uwsgi] +socket = /path/to/etebase_server.sock +chown-socket = EtebaseUser:www-data +chmod-socket = 660 +vacuum = true + + +uid = EtebaseUser +chdir = /path/to/etebase +home = %(chdir)/.venv +module = etebase_server.wsgi +master = true diff --git a/example-configs/nginx-uwsgi/etesync.ini b/example-configs/nginx-uwsgi/etesync.ini deleted file mode 100644 index e79eeee..0000000 --- a/example-configs/nginx-uwsgi/etesync.ini +++ /dev/null @@ -1,15 +0,0 @@ -# uwsgi configuration file -# typical location of this file would be /etc/uwsgi/sites/etesync.ini - -[uwsgi] -socket = /path/to/etesync_server.sock -chown-socket = EtesyncUser:www-data -chmod-socket = 660 -vacuum = true - - -uid = EtesyncUser -chdir = /path/to/etesync -home = %(chdir)/.venv -module = etesync_server.wsgi -master = true diff --git a/example-configs/nginx-uwsgi/my.server.name.conf b/example-configs/nginx-uwsgi/my.server.name.conf index b5b019d..6b5de6e 100644 --- a/example-configs/nginx-uwsgi/my.server.name.conf +++ b/example-configs/nginx-uwsgi/my.server.name.conf @@ -1,30 +1,36 @@ -# nginx configuration for etesync server running on https://my.server.name +# nginx configuration for etebase server running on https://my.server.name # typical location of this file would be /etc/nginx/sites-available/my.server.name.conf server { server_name my.server.name; - root /srv/http/etesync_server; + root /srv/http/etebase_server; + + client_max_body_size 20M; - client_max_body_size 5M; - location /static { expires 1y; try_files $uri $uri/ =404; } + location /media { + expires 1y; + try_files $uri $uri/ =404; + } + location / { - uwsgi_pass unix:/path/to/etesync_server.sock; + uwsgi_pass unix:/path/to/etebase_server.sock; include uwsgi_params; } # change 443 to say 9443 to run on a non standard port - listen 443 ssl; - listen [::]:443 ssl; + listen 443 ssl; + listen [::]:443 ssl; # Enable these two instead of the two above if your nginx supports http2 # listen 443 ssl http2; # listen [::]:443 ssl http2; - + ssl_certificate /path/to/certificate-file ssl_certificate_key /path/to/certificate-key-file # other ssl directives as needed +} diff --git a/example-configs/nginx-uwsgi/readme.md b/example-configs/nginx-uwsgi/readme.md deleted file mode 100644 index dad98b6..0000000 --- a/example-configs/nginx-uwsgi/readme.md +++ /dev/null @@ -1,20 +0,0 @@ -# Running `etesync` under `nginx` and `uwsgi` - -This configuration assumes that etesync server has been installed in the home folder of a non privileged user -called `EtesyncUser` following the instructions in . Also that static -files have been collected at `/srv/http/etesync_server` by running the following commands: - - sudo mkdir -p /srv/http/etesync_server/static - sudo chown -R EtesyncUser /srv/http/etesync_server - sudo su EtesyncUser - cd /path/to/etesync - ln -s /srv/http/etesync_server/static static - ./manage.py collectstatic - -It is also assumed that `nginx` and `uwsgi` have been installed system wide by `root`, and that `nginx` is running as user/group `www-data`. - -In this setup, `uwsgi` running as a `systemd` service as `root` creates a unix socket with read-write access -to both `EtesyncUser` and `nginx`. It then drops its `root` privilege and runs `etesync` as `EtesyncUser`. - -`nginx` listens on the `https` port (or a non standard port `https` port if desired), delivers static pages directly -and for everything else, communicates with `etesync` over the unix socket. From eac8fae376bd038fbd187965e289bb32cb652af5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 11 Sep 2020 16:01:44 +0300 Subject: [PATCH 276/511] README: update contribution information. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 33aaf64..1de61b6 100644 --- a/README.md +++ b/README.md @@ -119,4 +119,4 @@ You can now restart the server. # Supporting EteSync -Please consider registering an account even if you self-host in order to support the development of EteSync, or help by spreading the word. +Please consider registering an account even if you self-host in order to support the development of EteSync, or visit the [contribution](https://www.etesync.com/contribute/) for more information on how to support the service. From c04650f890005b9d1ba70cff1136afbb41531be8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 11 Sep 2020 16:02:47 +0300 Subject: [PATCH 277/511] README: update contribution information. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 781fd92..ee3447f 100644 --- a/README.md +++ b/README.md @@ -110,4 +110,4 @@ You can now restart the server. # Supporting Etebase -Please consider registering an account even if you self-host in order to support the development of Etebase, or help by spreading the word. +Please consider registering an account even if you self-host in order to support the development of Etebase, or visit the [contribution](https://www.etesync.com/contribute/) for more information on how to support the service. From 3de1d48b9ed43619c94e15d9f8b451e0d2e151b4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 13 Sep 2020 14:13:06 +0300 Subject: [PATCH 278/511] Browsable API: use input fields for relations. --- django_etebase/serializers.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 35b3562..b683580 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -187,6 +187,7 @@ class CollectionItemRevisionSerializer(BetterErrorsMixin, serializers.ModelSeria chunks = ChunksField( source='chunks_relation', queryset=models.RevisionChunkRelation.objects.all(), + style={'base_template': 'input.html'}, many=True ) meta = BinaryBase64Field() @@ -317,6 +318,7 @@ class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer) username = UserSlugRelatedField( source='user', read_only=True, + style={'base_template': 'input.html'}, ) class Meta: @@ -341,7 +343,8 @@ class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer) class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSerializer): username = UserSlugRelatedField( source='user', - queryset=User.objects + queryset=User.objects, + style={'base_template': 'input.html'}, ) collection = serializers.CharField(source='collection.uid') fromPubkey = BinaryBase64Field(source='fromMember.user.userinfo.pubkey', read_only=True) From 00cf2d83a05440f6166918bab55316d3e95fab03 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 13 Sep 2020 14:17:25 +0300 Subject: [PATCH 279/511] Only enable browsable API when debugging is on. The reason for that is that the API may expose data that shouldn't be exposed, such as the list of users on the service. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 636287e..c5482d6 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -91,7 +91,7 @@ def msgpack_decode(content): class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) - renderer_classes = [JSONRenderer, MessagePackRenderer, BrowsableAPIRenderer] + renderer_classes = [JSONRenderer, MessagePackRenderer] + [BrowsableAPIRenderer] if settings.DEBUG else [] parser_classes = [JSONParser, MessagePackParser, FormParser, MultiPartParser] stoken_id_fields = None From 374048f01329ad36cc42991c5ebcbeda35ce8308 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 13 Sep 2020 14:37:48 +0300 Subject: [PATCH 280/511] Fix disabling of browseable API when debug is off. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index c5482d6..971d0fe 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -91,7 +91,7 @@ def msgpack_decode(content): class BaseViewSet(viewsets.ModelViewSet): authentication_classes = tuple(app_settings.API_AUTHENTICATORS) permission_classes = tuple(app_settings.API_PERMISSIONS) - renderer_classes = [JSONRenderer, MessagePackRenderer] + [BrowsableAPIRenderer] if settings.DEBUG else [] + renderer_classes = [JSONRenderer, MessagePackRenderer] + ([BrowsableAPIRenderer] if settings.DEBUG else []) parser_classes = [JSONParser, MessagePackParser, FormParser, MultiPartParser] stoken_id_fields = None From 4dbdb3d7cfeaaa07776a49caa1a80167c767708d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 20 Sep 2020 19:33:55 +0300 Subject: [PATCH 281/511] Invitations: gracefully error when trying to invite an already invited user. --- django_etebase/serializers.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index b683580..3c32e10 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -17,7 +17,7 @@ import base64 from django.core.files.base import ContentFile from django.core import exceptions as django_exceptions from django.contrib.auth import get_user_model -from django.db import transaction +from django.db import IntegrityError, transaction from rest_framework import serializers, status from . import models from .utils import get_user_queryset, create_user @@ -368,7 +368,10 @@ class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSeriali member = collection.members.get(user=request.user) with transaction.atomic(): - return type(self).Meta.model.objects.create(**validated_data, fromMember=member) + try: + return type(self).Meta.model.objects.create(**validated_data, fromMember=member) + except IntegrityError: + raise EtebaseValidationError('invitation_exists', 'Invitation already exists') def update(self, instance, validated_data): with transaction.atomic(): From 7b8b0a568567b9046c1fbc32f73bedcae25039d5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 21 Sep 2020 12:09:19 +0300 Subject: [PATCH 282/511] Login: make case insensitive. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 971d0fe..0ba3b44 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -654,7 +654,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(data, status=status.HTTP_201_CREATED) def get_login_user(self, username): - kwargs = {User.USERNAME_FIELD: username.lower()} + kwargs = {User.USERNAME_FIELD + '__iexact': username.lower()} try: user = self.get_queryset().get(**kwargs) if not hasattr(user, 'userinfo'): From 18b3f45b79c01915310b619038afdf09721669ea Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 22 Sep 2020 11:33:17 +0300 Subject: [PATCH 283/511] Collection main_item: make a OneToOneField intsead of just a foreign key. --- .../migrations/0030_auto_20200922_0832.py | 19 +++++++++++++++++++ django_etebase/models.py | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 django_etebase/migrations/0030_auto_20200922_0832.py diff --git a/django_etebase/migrations/0030_auto_20200922_0832.py b/django_etebase/migrations/0030_auto_20200922_0832.py new file mode 100644 index 0000000..d5fa95d --- /dev/null +++ b/django_etebase/migrations/0030_auto_20200922_0832.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-09-22 08:32 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0029_auto_20200907_0801'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='main_item', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='parent', to='django_etebase.collectionitem'), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 797aa0f..d3405f1 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -28,7 +28,7 @@ UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{20,}$', message='Not a val class Collection(models.Model): - main_item = models.ForeignKey('CollectionItem', related_name='parent', null=True, on_delete=models.SET_NULL) + main_item = models.OneToOneField('CollectionItem', related_name='parent', null=True, on_delete=models.SET_NULL) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) def __str__(self): From 5d9b47531ba99a0ee1803d6818814c9a3d1206b8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 22 Sep 2020 12:17:33 +0300 Subject: [PATCH 284/511] Collectin: make sure collections always have a unique UID. --- django_etebase/models.py | 13 +++++++++++++ django_etebase/serializers.py | 2 ++ 2 files changed, 15 insertions(+) diff --git a/django_etebase/models.py b/django_etebase/models.py index d3405f1..914b763 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -21,7 +21,10 @@ from django.db.models import Q from django.utils.functional import cached_property from django.utils.crypto import get_random_string +from rest_framework import status + from . import app_settings +from .exceptions import EtebaseValidationError UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{20,}$', message='Not a valid UID') @@ -57,6 +60,16 @@ class Collection(models.Model): return stoken.uid + def validate_unique(self, exclude=None): + super().validate_unique(exclude=exclude) + if exclude is None or 'main_item' in exclude: + return + + if self.__class__.objects.filter(owner=self.owner, main_item__uid=self.main_item.uid) \ + .exclude(id=self.id).exists(): + raise EtebaseValidationError('unique_uid', 'Collection with this uid already exists', + status_code=status.HTTP_409_CONFLICT) + class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 3c32e10..d371f13 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -297,6 +297,8 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) instance.main_item = main_item + + instance.full_clean() instance.save() process_revisions_for_item(main_item, revision_data) From 5c803d8a51e5d02e3a321cc5de1ff68245f78df8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 22 Sep 2020 18:00:28 +0300 Subject: [PATCH 285/511] Only expose drf's auth in debug mode. --- etebase_server/urls.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/etebase_server/urls.py b/etebase_server/urls.py index 0c114af..fddc32f 100644 --- a/etebase_server/urls.py +++ b/etebase_server/urls.py @@ -1,3 +1,4 @@ +from django.conf import settings from django.conf.urls import include, url from django.contrib import admin from django.urls import path @@ -6,7 +7,11 @@ from django.views.generic import TemplateView urlpatterns = [ url(r'^api/', include('django_etebase.urls')), url(r'^admin/', admin.site.urls), - url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), path('', TemplateView.as_view(template_name='success.html')), ] + +if settings.DEBUG: + urlpatterns += [ + url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), + ] From f5ced873ac659b1992bf0f3f9bd2e3b3b0b95099 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 23 Sep 2020 16:27:20 +0300 Subject: [PATCH 286/511] Lint: fix lint errors. --- django_etebase/serializers.py | 6 ++++-- django_etebase/views.py | 7 +++---- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index d371f13..4518f58 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -220,7 +220,8 @@ class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): cur_etag = instance.etag if not created else None if validate_etag and cur_etag != etag: - raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(cur_etag, etag), status_code=status.HTTP_409_CONFLICT) + raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(cur_etag, etag), + status_code=status.HTTP_409_CONFLICT) if not created: # We don't have to use select_for_update here because the unique constraint on current guards against @@ -249,7 +250,8 @@ class CollectionItemDepSerializer(BetterErrorsMixin, serializers.ModelSerializer item = self.__class__.Meta.model.objects.get(uid=data['uid']) etag = data['etag'] if item.etag != etag: - raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(item.etag, etag), status_code=status.HTTP_409_CONFLICT) + raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(item.etag, etag), + status_code=status.HTTP_409_CONFLICT) return data diff --git a/django_etebase/views.py b/django_etebase/views.py index 0ba3b44..ec3b373 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -13,13 +13,12 @@ # along with this program. If not, see . import msgpack -from functools import reduce from django.conf import settings from django.contrib.auth import get_user_model, user_logged_in, user_logged_out from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError -from django.db.models import Max, Q, F, Value as V +from django.db.models import Max, Value as V from django.db.models.functions import Coalesce, Greatest from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -663,7 +662,6 @@ class AuthenticationViewSet(viewsets.ViewSet): except User.DoesNotExist: raise AuthenticationFailed({'code': 'user_not_found', 'detail': 'User not found'}) - def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): from datetime import datetime @@ -698,7 +696,8 @@ class AuthenticationViewSet(viewsets.ViewSet): try: verify_key.verify(response_raw, signature) except nacl.exceptions.BadSignatureError: - return Response({'code': 'login_bad_signature', 'detail': 'Wrong password for user.'}, status=status.HTTP_401_UNAUTHORIZED) + return Response({'code': 'login_bad_signature', 'detail': 'Wrong password for user.'}, + status=status.HTTP_401_UNAUTHORIZED) return None From 8a557ff82cd4b9147915f7aec7a7e8ae51dbcbf0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Sep 2020 09:42:01 +0300 Subject: [PATCH 287/511] Disable signups by default. The next commit includes README instructions on how to create users and enable signups. --- django_etebase/utils.py | 6 ++++++ etebase_server/settings.py | 3 ++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/django_etebase/utils.py b/django_etebase/utils.py index 08f81ae..1351f9b 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -1,4 +1,6 @@ from django.contrib.auth import get_user_model +from django.core.exceptions import PermissionDenied + from . import app_settings @@ -18,3 +20,7 @@ def create_user(*args, **kwargs): return custom_func(*args, **kwargs) _ = kwargs.pop('view') return User.objects.create_user(*args, **kwargs) + + +def create_user_blocked(*args, **kwargs): + raise PermissionDenied('Signup is disabled for this server. Please refer to the README for more information.') diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 7af0c60..f785cb7 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -29,7 +29,7 @@ AUTH_USER_MODEL = 'myauth.User' SECRET_FILE = os.path.join(BASE_DIR, "secret.txt") # SECURITY WARNING: don't run with debug turned on in production! -DEBUG = False +DEBUG = True ALLOWED_HOSTS = [] @@ -167,6 +167,7 @@ if any(os.path.isfile(x) for x in config_locations): ETEBASE_API_PERMISSIONS = ('rest_framework.permissions.IsAuthenticated', ) ETEBASE_API_AUTHENTICATORS = ('django_etebase.token_auth.authentication.TokenAuthentication', 'rest_framework.authentication.SessionAuthentication') +ETEBASE_CREATE_USER_FUNC = 'django_etebase.utils.create_user_blocked' # Make an `etebase_server_settings` module available to override settings. try: From 1e7e9eceacaae4a2c8c291c7d3c41227e1915925 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Sep 2020 09:45:31 +0300 Subject: [PATCH 288/511] README: update signup instructions to EteSync 2.0. Fixes #55. --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index ee3447f..80396b6 100644 --- a/README.md +++ b/README.md @@ -79,12 +79,12 @@ Create yourself an admin user: ./manage.py createsuperuser ``` -At this stage you can either just use the admin user, or better yet, go to: ```www.your-etesync-install.com/admin``` -and create a non-privileged user that you can use. +At this stage you need to create accounts to be used with the EteSync apps. To do that, please go to: +`www.your-etesync-install.com/admin` and create a new user to be used with the service. -That's it! - -Now all that's left is to open the EteSync app, add an account, and set your custom server address under the "advance" section. +After this user has been created, you can use any of the EteSync apps to signup (not login!) with the same username and +email in order to set up the account. Please make sure to click "advance" and set your customer server address when you +do. # `SECRET_KEY` and `secret.txt` From c9983fd79dc13a4a5b53ff31fe2533e00404312a Mon Sep 17 00:00:00 2001 From: Simon Vandevelde Date: Sun, 27 Sep 2020 16:48:52 +0200 Subject: [PATCH 289/511] Update README for Etebase with new wiki links (#56) --- README.md | 26 +++--- icon.svg | 241 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 256 insertions(+), 11 deletions(-) create mode 100644 icon.svg diff --git a/README.md b/README.md index 80396b6..6e28a43 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@

-

EteSync - Secure Data Sync

+

Etebase - Encrypt Everything

A skeleton app for running your own [Etebase](https://www.etebase.com) server @@ -9,7 +9,7 @@ A skeleton app for running your own [Etebase](https://www.etebase.com) server ## From source -Before installing the EteSync server make sure you install `virtualenv` (for **Python 3**): +Before installing the Etebase server make sure you install `virtualenv` (for **Python 3**): * Arch Linux: `pacman -S python-virtualenv` * Debian/Ubuntu: `apt-get install python3-virtualenv` @@ -18,9 +18,10 @@ Before installing the EteSync server make sure you install `virtualenv` (for **P Then just clone the git repo and set up this app: ``` -git clone https://github.com/etesync/server-skeleton.git +git clone https://github.com/etesync/server.git etebase -cd server-skeleton +cd etebase +git checkout etebase # Set up the environment and deps virtualenv -p python3 venv # If doesn't work, try: virtualenv3 venv @@ -32,8 +33,11 @@ pip install -r requirements.txt # Configuration If you are familiar with Django you can just edit the [settings file](etesync_server/settings.py) -according to the [Django deployment checklist](https://docs.djangoproject.com/en/dev/howto/deployment/checklist/) -if you are not, we will soon provide a simple configuration file for easy deployment like we had with EteSync. +according to the [Django deployment checklist](https://docs.djangoproject.com/en/dev/howto/deployment/checklist/). +If you are not, we also provide a simple [configuration file](https://github.com/etesync/server/blob/etebase/etebase-server.ini.example) for easy deployment which you can use. +To use the easy configuration file rename it to `etebase-server.ini` and place it either at the root of this repository or in `/etc/etebase-server`. + +There is also a [wikipage](https://github.com/etesync/server/wiki/Basic-Setup-Etebase-(EteSync-v2)) detailing this basic setup. Some particular settings that should be edited are: * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) @@ -46,7 +50,7 @@ will be served generation purposes. See below for how default configuration of `SECRET_KEY` works for this project. -Now you can initialise our django app +Now you can initialise our django app. ``` ./manage.py migrate @@ -62,14 +66,14 @@ Using the debug server in production is not recommended, so please read the foll # Production deployment -EteSync is based on Django so you should refer to one of the following +There are more details about a proper production setup using Daphne and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-Daphne-and-Nginx). + +Etebase is based on Django so you should refer to one of the following * The instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). * Instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). -There are more details about a proper production setup using uWSGI and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-uWSGI-and-Nginx). - The webserver should also be configured to serve Etebase using TLS. -A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-EteSync) as well. +A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-Etebase) as well. # Usage diff --git a/icon.svg b/icon.svg new file mode 100644 index 0000000..4827d1d --- /dev/null +++ b/icon.svg @@ -0,0 +1,241 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From 62146881703a0c493f82367ac065395919e5b468 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Sep 2020 14:56:29 +0300 Subject: [PATCH 290/511] Invitations: share the username of the inviter. --- django_etebase/serializers.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 4518f58..4798f97 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -351,12 +351,14 @@ class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSeriali style={'base_template': 'input.html'}, ) collection = serializers.CharField(source='collection.uid') + fromUsername = BinaryBase64Field(source='fromMember.user.username', read_only=True) fromPubkey = BinaryBase64Field(source='fromMember.user.userinfo.pubkey', read_only=True) signedEncryptionKey = BinaryBase64Field() class Meta: model = models.CollectionInvitation - fields = ('username', 'uid', 'collection', 'signedEncryptionKey', 'accessLevel', 'fromPubkey', 'version') + fields = ('username', 'uid', 'collection', 'signedEncryptionKey', 'accessLevel', + 'fromUsername', 'fromPubkey', 'version') def validate_user(self, value): request = self.context['request'] From 06f2dd72a7458769355f48516f3f3feda4d53562 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 1 Oct 2020 16:45:47 +0300 Subject: [PATCH 291/511] Exception: fix detail/code for exception. --- django_etebase/views.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index ec3b373..31e914a 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -574,7 +574,8 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): return Response(content, status=status.HTTP_400_BAD_REQUEST) if not permissions.is_collection_admin(collection, request.user): - raise PermissionDenied('User is not an admin of this collection') + raise PermissionDenied({'code': 'admin_access_required', + 'detail': 'User is not an admin of this collection'}) serializer.save(collection=collection) From 9152e6f42d4afe8166b864170ea708f071441639 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 8 Oct 2020 21:01:45 +0300 Subject: [PATCH 292/511] Fix bad stoken error. We were calling the validation constructor wrong. --- django_etebase/views.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 31e914a..6003d4b 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -116,11 +116,7 @@ class BaseViewSet(viewsets.ModelViewSet): try: return Stoken.objects.get(uid=stoken) except Stoken.DoesNotExist: - raise EtebaseValidationError({ - 'code': 'bad_stoken', - 'detail': 'Invalid stoken.', - }, - status_code=status.HTTP_400_BAD_REQUEST) + raise EtebaseValidationError('bad_stoken', 'Invalid stoken.', status_code=status.HTTP_400_BAD_REQUEST) return None From 74f40abc65567674448bb5c4dfec4cc835d1a523 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 8 Oct 2020 21:03:54 +0300 Subject: [PATCH 293/511] Account: add a dashboard url endpoint. This lets servers share a dashboard url with clients so that they in turn can present clients with a settings dashboard. We currently use it on the main server, but self-hosted servers may also benefit from it for letting users manage some of their settings (e.g. 2FA). --- django_etebase/app_settings.py | 7 +++++++ django_etebase/views.py | 12 ++++++++++++ 2 files changed, 19 insertions(+) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index 33dc65f..3c580b2 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -61,6 +61,13 @@ class AppSettings: return self.import_from_str(func) return None + @cached_property + def DASHBOARD_URL_FUNC(self): # pylint: disable=invalid-name + func = self._setting("DASHBOARD_URL_FUNC", None) + if func is not None: + return self.import_from_str(func) + return None + @cached_property def CHUNK_PATH_FUNC(self): # pylint: disable=invalid-name func = self._setting("CHUNK_PATH_FUNC", None) diff --git a/django_etebase/views.py b/django_etebase/views.py index 6003d4b..d421e43 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -783,6 +783,18 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_200_OK) + @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) + def dashboard_url(self, request, *args, **kwargs): + get_dashboard_url = app_settings.DASHBOARD_URL_FUNC + if get_dashboard_url is None: + raise EtebaseValidationError('not_supported', 'This server doesn\'t have a user dashboard.', + status_code=status.HTTP_400_BAD_REQUEST) + + ret = { + 'url': get_dashboard_url(request, *args, **kwargs), + } + return Response(ret) + class TestAuthenticationViewSet(viewsets.ViewSet): allowed_methods = ['POST'] From 9cad5d62e1518502ca149d77a8c254fba98eff89 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 9 Oct 2020 13:10:41 +0300 Subject: [PATCH 294/511] Account: change Dashboard URL endpoint's permissions. We only want to require that the account is authenticated, not the rest of the permissions. As we want to be able to get a dashboard url for accounts that aren't currently valid. --- django_etebase/views.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index d421e43..8c55366 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -30,6 +30,7 @@ from rest_framework.response import Response from rest_framework.parsers import JSONParser, FormParser, MultiPartParser from rest_framework.renderers import BrowsableAPIRenderer from rest_framework.exceptions import AuthenticationFailed +from rest_framework.permissions import IsAuthenticated import nacl.encoding import nacl.signing @@ -783,7 +784,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_200_OK) - @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) + @action_decorator(detail=False, methods=['POST'], permission_classes=[IsAuthenticated]) def dashboard_url(self, request, *args, **kwargs): get_dashboard_url = app_settings.DASHBOARD_URL_FUNC if get_dashboard_url is None: From 24c161b0d84d59572ec342631416607ebc197e0e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 11:09:22 +0300 Subject: [PATCH 295/511] Signup: don't try to clean fields for objects we haven't created. --- django_etebase/serializers.py | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 4798f97..5cc24aa 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -462,19 +462,17 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): # Create the user and save the casing the user chose as the first name try: instance = create_user(**user_data, password=None, first_name=user_data['username'], view=view) + instance.clean_fields() except EtebaseValidationError as e: raise e + except django_exceptions.ValidationError as e: + self.transform_validation_error("user", e) except Exception as e: raise EtebaseValidationError('generic', str(e)) if hasattr(instance, 'userinfo'): raise EtebaseValidationError('user_exists', 'User already exists', status_code=status.HTTP_409_CONFLICT) - try: - instance.clean_fields() - except django_exceptions.ValidationError as e: - self.transform_validation_error("user", e) - models.UserInfo.objects.create(**validated_data, owner=instance) return instance From 47f3e088464bdc623a41139331085e3f2026e284 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 11:10:55 +0300 Subject: [PATCH 296/511] Signup: improve docs. --- django_etebase/serializers.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 5cc24aa..759536b 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -443,6 +443,9 @@ class UserSignupSerializer(BetterErrorsMixin, serializers.ModelSerializer): class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): + """Used both for creating new accounts and setting up existing ones for the first time. + When setting up existing ones the email is ignored." + """ user = UserSignupSerializer(many=False) salt = BinaryBase64Field() loginPubkey = BinaryBase64Field() From c7bd01b2d12ac437ddab95283fcd2eb8085ea5a4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 12:09:29 +0300 Subject: [PATCH 297/511] Logout: allow any authenticated user (instead of normal permissions). We should always allow users to log out if they are authenticated. This doesn't need to use the global permissions. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 8c55366..2d9b76c 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -756,7 +756,7 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(data, status=status.HTTP_200_OK) - @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) + @action_decorator(detail=False, methods=['POST'], permission_classes=[IsAuthenticated]) def logout(self, request, *args, **kwargs): request.auth.delete() user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) From aa7b049b62e42316d0a342b3b6edbe066761f53b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 13:37:05 +0300 Subject: [PATCH 298/511] Stoken: always return the stoken object, not the rev. --- django_etebase/views.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 2d9b76c..897bcb5 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -138,15 +138,14 @@ class BaseViewSet(viewsets.ModelViewSet): for row in queryset: rowmaxid = getattr(row, 'max_stoken') or -1 maxid = max(maxid, rowmaxid) - new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid).uid + new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid) - return new_stoken + return new_stoken or None def filter_by_stoken_and_limit(self, request, queryset): limit = int(request.GET.get('limit', 50)) queryset, stoken_rev = self.filter_by_stoken(request, queryset) - stoken = stoken_rev.uid if stoken_rev is not None else None result = list(queryset[:limit + 1]) if len(result) < limit + 1: @@ -155,9 +154,9 @@ class BaseViewSet(viewsets.ModelViewSet): done = False result = result[:-1] - new_stoken = self.get_queryset_stoken(result) or stoken + new_stoken_obj = self.get_queryset_stoken(result) or stoken_rev - return result, new_stoken, done + return result, new_stoken_obj, done # Change how our list works by default def list(self, request, collection_uid=None, *args, **kwargs): @@ -211,7 +210,8 @@ class CollectionViewSet(BaseViewSet): def list(self, request, *args, **kwargs): queryset = self.get_queryset() - result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) + result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) + new_stoken = new_stoken_obj and new_stoken_obj.uid serializer = self.get_serializer(result, many=True) @@ -278,7 +278,8 @@ class CollectionItemViewSet(BaseViewSet): if not self.request.query_params.get('withCollection', False): queryset = queryset.filter(parent__isnull=True) - result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) + result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) + new_stoken = new_stoken_obj and new_stoken_obj.uid serializer = self.get_serializer(result, many=True) @@ -342,8 +343,9 @@ class CollectionItemViewSet(BaseViewSet): revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) - new_stoken = self.get_queryset_stoken(queryset) - stoken = getattr(stoken_rev, 'uid', None) if stoken_rev is not None else None + new_stoken_obj = self.get_queryset_stoken(queryset) + new_stoken = new_stoken_obj and new_stoken_obj.uid + stoken = stoken_rev and getattr(stoken_rev, 'uid', None) new_stoken = new_stoken or stoken serializer = self.get_serializer(queryset, many=True) @@ -481,7 +483,8 @@ class CollectionMemberViewSet(BaseViewSet): def list(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset().order_by('id') - result, new_stoken, done = self.filter_by_stoken_and_limit(request, queryset) + result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) + new_stoken = new_stoken_obj and new_stoken_obj.uid serializer = self.get_serializer(result, many=True) ret = { From 741b6d7c52dbe785dd3d5e94025a7c1dc23f6dff Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 13:29:29 +0300 Subject: [PATCH 299/511] Collection removed memberships: only return removed memberships within our returned range. Before this change we were returning all of the removed memberships that happened after stoken. Though instead, we should just return the removed memberships that happened after stoken and before the new stoken we are returning. --- django_etebase/views.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 897bcb5..c7162dd 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -224,8 +224,13 @@ class CollectionViewSet(BaseViewSet): stoken_obj = self.get_stoken_obj(request) if stoken_obj is not None: # FIXME: honour limit? (the limit should be combined for data and this because of stoken) - remed = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) \ - .values_list('collection__main_item__uid', flat=True) + remed_qs = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) + if not ret['done']: + # We only filter by the new_stoken if we are not done. This is because if we are done, the new stoken + # can point to the most recent collection change rather than most recent removed membership. + remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) + + remed = remed_qs.values_list('collection__main_item__uid', flat=True) if len(remed) > 0: ret['removedMemberships'] = [{'uid': x} for x in remed] From acd22b9b47d7a899e676c921abfec81f86dd8533 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 16:30:16 +0300 Subject: [PATCH 300/511] Serializers: remove unused field. --- django_etebase/serializers.py | 8 -------- 1 file changed, 8 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 759536b..fe9ae11 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -86,14 +86,6 @@ class CollectionEncryptionKeyField(BinaryBase64Field): return None -class CollectionContentField(BinaryBase64Field): - def get_attribute(self, instance): - request = self.context.get('request', None) - if request is not None: - return instance.members.get(user=request.user).encryptionKey - return None - - class UserSlugRelatedField(serializers.SlugRelatedField): def get_queryset(self): view = self.context.get('view', None) From 5d8a92f0001c6595826c2def2af25d509c69ef52 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 13 Oct 2020 17:13:07 +0300 Subject: [PATCH 301/511] Collections: add support for collection types. We also added the field for invitations, as it's needed for collections to work. --- .../migrations/0031_auto_20201013_1336.py | 29 ++++++++++++++++ .../migrations/0032_auto_20201013_1409.py | 18 ++++++++++ django_etebase/models.py | 6 ++++ django_etebase/serializers.py | 34 +++++++++++++++++-- django_etebase/views.py | 16 +++++++++ 5 files changed, 100 insertions(+), 3 deletions(-) create mode 100644 django_etebase/migrations/0031_auto_20201013_1336.py create mode 100644 django_etebase/migrations/0032_auto_20201013_1409.py diff --git a/django_etebase/migrations/0031_auto_20201013_1336.py b/django_etebase/migrations/0031_auto_20201013_1336.py new file mode 100644 index 0000000..ca45dd4 --- /dev/null +++ b/django_etebase/migrations/0031_auto_20201013_1336.py @@ -0,0 +1,29 @@ +# Generated by Django 3.1.1 on 2020-10-13 13:36 + +from django.conf import settings +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ('django_etebase', '0030_auto_20200922_0832'), + ] + + operations = [ + migrations.CreateModel( + name='CollectionType', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('uid', models.BinaryField(db_index=True, editable=True)), + ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ], + ), + migrations.AddField( + model_name='collectionmember', + name='collectionType', + field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.collectiontype'), + ), + ] diff --git a/django_etebase/migrations/0032_auto_20201013_1409.py b/django_etebase/migrations/0032_auto_20201013_1409.py new file mode 100644 index 0000000..5594006 --- /dev/null +++ b/django_etebase/migrations/0032_auto_20201013_1409.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.1 on 2020-10-13 14:09 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0031_auto_20201013_1336'), + ] + + operations = [ + migrations.AlterField( + model_name='collectiontype', + name='uid', + field=models.BinaryField(db_index=True, editable=True, unique=True), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 914b763..0036884 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -30,6 +30,11 @@ from .exceptions import EtebaseValidationError UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{20,}$', message='Not a valid UID') +class CollectionType(models.Model): + owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True) + + class Collection(models.Model): main_item = models.OneToOneField('CollectionItem', related_name='parent', null=True, on_delete=models.SET_NULL) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) @@ -162,6 +167,7 @@ class CollectionMember(models.Model): collection = models.ForeignKey(Collection, related_name='members', on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) + collectionType = models.ForeignKey(CollectionType, on_delete=models.PROTECT, null=True) accessLevel = models.IntegerField( choices=AccessLevels.choices, default=AccessLevels.READ_ONLY, diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index fe9ae11..58ad10c 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -86,6 +86,15 @@ class CollectionEncryptionKeyField(BinaryBase64Field): return None +class CollectionTypeField(BinaryBase64Field): + def get_attribute(self, instance): + request = self.context.get('request', None) + if request is not None: + collection_type = instance.members.get(user=request.user).collectionType + return collection_type and collection_type.uid + return None + + class UserSlugRelatedField(serializers.SlugRelatedField): def get_queryset(self): view = self.context.get('view', None) @@ -256,8 +265,15 @@ class CollectionItemBulkGetSerializer(BetterErrorsMixin, serializers.ModelSerial fields = ('uid', 'etag') +class CollectionListMultiSerializer(BetterErrorsMixin, serializers.Serializer): + collectionTypes = serializers.ListField( + child=BinaryBase64Field() + ) + + class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): collectionKey = CollectionEncryptionKeyField() + collectionType = CollectionTypeField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) @@ -265,7 +281,7 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): class Meta: model = models.Collection - fields = ('item', 'accessLevel', 'collectionKey', 'stoken') + fields = ('item', 'accessLevel', 'collectionKey', 'collectionType', 'stoken') def get_access_level_from_context(self, obj): request = self.context.get('request', None) @@ -276,6 +292,7 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" collection_key = validated_data.pop('collectionKey') + collection_type = validated_data.pop('collectionType') main_item_data = validated_data.pop('main_item') etag = main_item_data.pop('etag') @@ -297,11 +314,16 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): process_revisions_for_item(main_item, revision_data) + user = validated_data.get('owner') + + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) + models.CollectionMember(collection=instance, stoken=models.Stoken.objects.create(), - user=validated_data.get('owner'), + user=user, accessLevel=models.AccessLevels.ADMIN, encryptionKey=collection_key, + collectionType=collection_type_obj, ).save() return instance @@ -381,6 +403,7 @@ class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSeriali class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): + collectionType = BinaryBase64Field() encryptionKey = BinaryBase64Field() def create(self, validated_data): @@ -388,13 +411,18 @@ class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): with transaction.atomic(): invitation = self.context['invitation'] encryption_key = validated_data.get('encryptionKey') + collection_type = validated_data.pop('collectionType') + + user = invitation.user + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) member = models.CollectionMember.objects.create( collection=invitation.collection, stoken=models.Stoken.objects.create(), - user=invitation.user, + user=user, accessLevel=invitation.accessLevel, encryptionKey=encryption_key, + collectionType=collection_type_obj, ) models.CollectionMemberRemoved.objects.filter( diff --git a/django_etebase/views.py b/django_etebase/views.py index c7162dd..34f6254 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -66,6 +66,7 @@ from .serializers import ( CollectionItemDepSerializer, CollectionItemRevisionSerializer, CollectionItemChunkSerializer, + CollectionListMultiSerializer, CollectionMemberSerializer, CollectionInvitationSerializer, InvitationAcceptSerializer, @@ -210,6 +211,21 @@ class CollectionViewSet(BaseViewSet): def list(self, request, *args, **kwargs): queryset = self.get_queryset() + return self.list_common(request, queryset, *args, **kwargs) + + @action_decorator(detail=False, methods=['POST']) + def list_multi(self, request, *args, **kwargs): + serializer = CollectionListMultiSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + + collection_types = serializer.validated_data['collectionTypes'] + + queryset = self.get_queryset() + queryset = queryset.filter(members__collectionType__uid__in=collection_types) + + return self.list_common(request, queryset, *args, **kwargs) + + def list_common(self, request, queryset, *args, **kwargs): result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) new_stoken = new_stoken_obj and new_stoken_obj.uid From 409248d419aa0fbbc23fad4ec163c95fb59ebad5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 15 Oct 2020 10:50:07 +0300 Subject: [PATCH 302/511] CollectionTypes: add backward compatibility adjustments until 2.0 is out. --- django_etebase/serializers.py | 3 ++- django_etebase/views.py | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 58ad10c..d753bf9 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -273,7 +273,8 @@ class CollectionListMultiSerializer(BetterErrorsMixin, serializers.Serializer): class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): collectionKey = CollectionEncryptionKeyField() - collectionType = CollectionTypeField() + # FIXME: make required once "collection-type-migration" is done + collectionType = CollectionTypeField(required=False) accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) diff --git a/django_etebase/views.py b/django_etebase/views.py index 34f6254..5328c84 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -18,7 +18,7 @@ from django.conf import settings from django.contrib.auth import get_user_model, user_logged_in, user_logged_out from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError -from django.db.models import Max, Value as V +from django.db.models import Max, Value as V, Q from django.db.models.functions import Coalesce, Greatest from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -221,7 +221,9 @@ class CollectionViewSet(BaseViewSet): collection_types = serializer.validated_data['collectionTypes'] queryset = self.get_queryset() - queryset = queryset.filter(members__collectionType__uid__in=collection_types) + # FIXME: Remove the isnull part once "collection-type-migration" is done + queryset = queryset.filter( + Q(members__collectionType__uid__in=collection_types) | Q(members__collectionType__isnull=True)) return self.list_common(request, queryset, *args, **kwargs) From 5bce4d9932960e45b9ebefff94cfc1553e63761f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 15 Oct 2020 15:00:20 +0300 Subject: [PATCH 303/511] Collection Type: fix backwards compatibility for creating new collections. Continuation to 409248d419aa0fbbc23fad4ec163c95fb59ebad5. --- django_etebase/serializers.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index d753bf9..038e879 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -293,7 +293,8 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" collection_key = validated_data.pop('collectionKey') - collection_type = validated_data.pop('collectionType') + # FIXME: remove the None fallback once "collection-type-migration" is done + collection_type = validated_data.pop('collectionType', None) main_item_data = validated_data.pop('main_item') etag = main_item_data.pop('etag') @@ -317,7 +318,11 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): user = validated_data.get('owner') - collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) + # FIXME: remove the if statement (and else branch) once "collection-type-migration" is done + if collection_type is not None: + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) + else: + collection_type_obj = None models.CollectionMember(collection=instance, stoken=models.Stoken.objects.create(), From 529b5c22e8de679f1bed8693acd9a78fffcccff6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 18 Oct 2020 18:13:38 +0300 Subject: [PATCH 304/511] README: mention that this is EteSync 2.0 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6e28a43..1dcc8f3 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@

Etebase - Encrypt Everything

-A skeleton app for running your own [Etebase](https://www.etebase.com) server +A skeleton app for running your own [Etebase](https://www.etebase.com) (EteSync 2.0) server. # Installation From 3b4cd424ab39721d3f2036fab46ab47ea1776ddd Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 18 Oct 2020 18:28:51 +0300 Subject: [PATCH 305/511] Update changelog. --- ChangeLog.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 ChangeLog.md diff --git a/ChangeLog.md b/ChangeLog.md new file mode 100644 index 0000000..9c21c49 --- /dev/null +++ b/ChangeLog.md @@ -0,0 +1,4 @@ +# Changelog + +## Version 0.5.0 +* First Etebase-server release (was EteSync-server before) From 0a246aaa8d22a78542fc7e295ffea5c5ea19cff9 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 21 Oct 2020 18:45:09 +0300 Subject: [PATCH 306/511] README: improve self-hosting instructions --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 1dcc8f3..5509f87 100644 --- a/README.md +++ b/README.md @@ -84,11 +84,12 @@ Create yourself an admin user: ``` At this stage you need to create accounts to be used with the EteSync apps. To do that, please go to: -`www.your-etesync-install.com/admin` and create a new user to be used with the service. +`www.your-etesync-install.com/admin` and create a new user to be used with the service. Do *not* set +a password for the user, as Etebase uses a zero-knowledge proof for authentication. -After this user has been created, you can use any of the EteSync apps to signup (not login!) with the same username and -email in order to set up the account. Please make sure to click "advance" and set your customer server address when you -do. +After this user has been created, you can use any of the EteSync apps to signup (or login) with the same username and +email in order to set up the account. The password used at that point will be used to setup the account. +Don't forget to set your custom server address under "Advanced". # `SECRET_KEY` and `secret.txt` From 037f0f79a753ccabf60e7dc21ac145615641817c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 24 Oct 2020 10:38:48 +0300 Subject: [PATCH 307/511] README: improve the update instructions for versions < 0.5.0 --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index 5509f87..e112bf2 100644 --- a/README.md +++ b/README.md @@ -106,6 +106,8 @@ for more information about the `SECRET_KEY` and its uses. # Updating +## Updating from version 0.5.0 onwards + First, run `git pull --rebase` to update this repository. Then, inside the virtualenv: 1. Run `pip install -U -r requirements.txt` to update the dependencies. @@ -113,6 +115,17 @@ Then, inside the virtualenv: You can now restart the server. +## Updating from version 0.5.0 or before + +The 0.5.0 release marks the change to the EteSync 2.0 protocol. EteSync 2.0 accounts are substantially different to 1.0 accounts, and require additional upgrade steps. In addition, the servers are incompatible, so 0.5.0 requires a fresh installation. + +Here are the update steps: +1. Chose any of the [the migration tools](https://www.etesync.com/user-guide/migrate-v2/) and make sure the underlying apps are up to date with all of your data. So for example, if you are using the Android client, make sure to sync before commencing. +2. Install the 0.5.0 version to a new path (you can't reuse the same database). +3. Run the 0.5.0 account and create the appropriate users as described in the installation/upgrade steps above. +4. Run the migration tool to migrate all of your data. +5. Add your new EteSync 2.0 accounst to all of your devices. + # Supporting Etebase Please consider registering an account even if you self-host in order to support the development of Etebase, or visit the [contribution](https://www.etesync.com/contribute/) for more information on how to support the service. From 46abeac2c051050d5400238347cf6c11008fc91b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 25 Oct 2020 12:14:55 +0200 Subject: [PATCH 308/511] Test reset: also reset memberships. --- django_etebase/views.py | 1 + 1 file changed, 1 insertion(+) diff --git a/django_etebase/views.py b/django_etebase/views.py index 5328c84..3c40e17 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -861,6 +861,7 @@ class TestAuthenticationViewSet(viewsets.ViewSet): # Delete all of the journal data for this user for a clear test env user.collection_set.all().delete() + user.collectionmember_set.all().delete() user.incoming_invitations.all().delete() # FIXME: also delete chunk files!!! From b097f3b8faaf5264a8c1c114681890b94c88e16e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 30 Oct 2020 16:46:21 +0200 Subject: [PATCH 309/511] README: update instructions and fix type Fixes #60 --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e112bf2..0ecf52a 100644 --- a/README.md +++ b/README.md @@ -84,8 +84,10 @@ Create yourself an admin user: ``` At this stage you need to create accounts to be used with the EteSync apps. To do that, please go to: -`www.your-etesync-install.com/admin` and create a new user to be used with the service. Do *not* set -a password for the user, as Etebase uses a zero-knowledge proof for authentication. +`www.your-etesync-install.com/admin` and create a new user to be used with the service. Set a random +password for the user such as `j3PmCRftyQMtM3eWvi8f`. No need to remember it, as it won't be used. +Etebase uses a zero-knowledge proof for authentication, so the user will just create a password when +creating the account from the apps. After this user has been created, you can use any of the EteSync apps to signup (or login) with the same username and email in order to set up the account. The password used at that point will be used to setup the account. @@ -124,7 +126,7 @@ Here are the update steps: 2. Install the 0.5.0 version to a new path (you can't reuse the same database). 3. Run the 0.5.0 account and create the appropriate users as described in the installation/upgrade steps above. 4. Run the migration tool to migrate all of your data. -5. Add your new EteSync 2.0 accounst to all of your devices. +5. Add your new EteSync 2.0 accounts to all of your devices. # Supporting Etebase From f55ebeae7c8061fcf04465cb46614b967044616f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 1 Nov 2020 10:40:47 +0200 Subject: [PATCH 310/511] Collection saving: add another verification for collection UID uniqueness. Even with the previous check, there could still be a race condition where two collections with the same UID are created. Adding this extra check after will prevent that from happening. --- django_etebase/serializers.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 038e879..1634d9f 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -296,6 +296,7 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): # FIXME: remove the None fallback once "collection-type-migration" is done collection_type = validated_data.pop('collectionType', None) + user = validated_data.get('owner') main_item_data = validated_data.pop('main_item') etag = main_item_data.pop('etag') revision_data = main_item_data.pop('content') @@ -303,6 +304,7 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): + _ = self.__class__.Meta.model.objects.select_for_update().filter(owner=user) if etag is not None: raise EtebaseValidationError('bad_etag', 'etag is not null') @@ -316,8 +318,6 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): process_revisions_for_item(main_item, revision_data) - user = validated_data.get('owner') - # FIXME: remove the if statement (and else branch) once "collection-type-migration" is done if collection_type is not None: collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) From 422b62d5b2ef184fe115c6e08f9cbbd733a858e5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 1 Nov 2020 11:17:44 +0200 Subject: [PATCH 311/511] Disallow creating new collections without a collection type set. --- django_etebase/serializers.py | 12 +++--------- django_etebase/views.py | 2 +- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 1634d9f..97dcd64 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -273,8 +273,7 @@ class CollectionListMultiSerializer(BetterErrorsMixin, serializers.Serializer): class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): collectionKey = CollectionEncryptionKeyField() - # FIXME: make required once "collection-type-migration" is done - collectionType = CollectionTypeField(required=False) + collectionType = CollectionTypeField() accessLevel = serializers.SerializerMethodField('get_access_level_from_context') stoken = serializers.CharField(read_only=True) @@ -293,8 +292,7 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" collection_key = validated_data.pop('collectionKey') - # FIXME: remove the None fallback once "collection-type-migration" is done - collection_type = validated_data.pop('collectionType', None) + collection_type = validated_data.pop('collectionType') user = validated_data.get('owner') main_item_data = validated_data.pop('main_item') @@ -318,11 +316,7 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): process_revisions_for_item(main_item, revision_data) - # FIXME: remove the if statement (and else branch) once "collection-type-migration" is done - if collection_type is not None: - collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) - else: - collection_type_obj = None + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) models.CollectionMember(collection=instance, stoken=models.Stoken.objects.create(), diff --git a/django_etebase/views.py b/django_etebase/views.py index 3c40e17..a60346b 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -221,7 +221,7 @@ class CollectionViewSet(BaseViewSet): collection_types = serializer.validated_data['collectionTypes'] queryset = self.get_queryset() - # FIXME: Remove the isnull part once "collection-type-migration" is done + # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( Q(members__collectionType__uid__in=collection_types) | Q(members__collectionType__isnull=True)) From a6f5e070a4d415cbc0a6a3a02348f1aabe5c9168 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 1 Nov 2020 11:32:16 +0200 Subject: [PATCH 312/511] Update changelog. --- ChangeLog.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index 9c21c49..8845089 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,4 +1,8 @@ # Changelog +## Version 0.5.1 +* Enforce collections to always have a collection type set +* Collection saving: add another verification for collection UID uniqueness. + ## Version 0.5.0 * First Etebase-server release (was EteSync-server before) From 843b59a0ac1e2076a514b9c52fc5c6941b27dff0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 2 Nov 2020 10:16:04 +0200 Subject: [PATCH 313/511] Login/Changepassword: change to verifying the hostname without the part. Verifying the port was causing issues, and anyhow, this check is paranoid and isn't strictly necessary for security. The problem is that Django's `get_host()` and the equivalent on some platforms returns it without the port, though on others (like e.g. the library we use from JS) it returns with the port. This was inconsistent and was causing authentication to fail. We thus relaxed the test to not include the port when matching, which should make it work consistently across all platforms. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index a60346b..2dc7adf 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -710,7 +710,7 @@ class AuthenticationViewSet(viewsets.ViewSet): elif challenge_data['userId'] != user.id: content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host != request.get_host(): + elif not settings.DEBUG and host.split(':', 1)[0] != request.get_host(): detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) content = {'code': 'wrong_host', 'detail': detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) From 65319047b82bd0a75f74352e4051ea8f4605e5d5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 5 Nov 2020 16:49:05 +0200 Subject: [PATCH 314/511] Remove leftover files from EteSync 1.0. No idea how they got here. --- etesync-server.ini.example | 15 --- etesync_server/__init__.py | 14 --- etesync_server/settings.py | 185 ------------------------------------- etesync_server/urls.py | 70 -------------- etesync_server/utils.py | 25 ----- etesync_server/wsgi.py | 16 ---- 6 files changed, 325 deletions(-) delete mode 100644 etesync-server.ini.example delete mode 100644 etesync_server/__init__.py delete mode 100644 etesync_server/settings.py delete mode 100644 etesync_server/urls.py delete mode 100644 etesync_server/utils.py delete mode 100644 etesync_server/wsgi.py diff --git a/etesync-server.ini.example b/etesync-server.ini.example deleted file mode 100644 index 424a26d..0000000 --- a/etesync-server.ini.example +++ /dev/null @@ -1,15 +0,0 @@ -[global] -secret_file = secret.txt -debug = false -;Advanced options, only uncomment if you know what you're doing: -;static_root = /path/to/static -;static_url = /static/ -;language_code = en-us -;time_zone = UTC - -[allowed_hosts] -allowed_host1 = example.com - -[database] -engine = django.db.backends.sqlite3 -name = db.sqlite3 diff --git a/etesync_server/__init__.py b/etesync_server/__init__.py deleted file mode 100644 index 227e8c9..0000000 --- a/etesync_server/__init__.py +++ /dev/null @@ -1,14 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - diff --git a/etesync_server/settings.py b/etesync_server/settings.py deleted file mode 100644 index cca7915..0000000 --- a/etesync_server/settings.py +++ /dev/null @@ -1,185 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -""" -Django settings for etesync_server project. - -Generated by 'django-admin startproject' using Django 2.0.5. - -For more information on this file, see -https://docs.djangoproject.com/en/2.0/topics/settings/ - -For the full list of settings and their values, see -https://docs.djangoproject.com/en/2.0/ref/settings/ -""" - -import os -import configparser -from .utils import get_secret_from_file - -# Build paths inside the project like this: os.path.join(BASE_DIR, ...) -BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - - -# Quick-start development settings - unsuitable for production -# See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/ - -# SECURITY WARNING: keep the secret key used in production secret! -# See secret.py for how this is generated; uses a file 'secret.txt' in the root -# directory -SECRET_FILE = os.path.join(BASE_DIR, "secret.txt") - -# SECURITY WARNING: don't run with debug turned on in production! -DEBUG = False - -ALLOWED_HOSTS = [] - -# Database -# https://docs.djangoproject.com/en/2.0/ref/settings/#databases - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': os.environ.get('ETESYNC_DB_PATH', - os.path.join(BASE_DIR, 'db.sqlite3')), - } -} - - -# Application definition - -INSTALLED_APPS = [ - 'django.contrib.admin', - 'django.contrib.auth', - 'django.contrib.contenttypes', - 'django.contrib.sessions', - 'django.contrib.messages', - 'django.contrib.staticfiles', - 'corsheaders', - 'rest_framework', - 'rest_framework.authtoken', - 'journal.apps.JournalConfig', -] - -MIDDLEWARE = [ - 'django.middleware.security.SecurityMiddleware', - 'django.contrib.sessions.middleware.SessionMiddleware', - 'corsheaders.middleware.CorsMiddleware', - 'django.middleware.common.CommonMiddleware', - 'django.middleware.csrf.CsrfViewMiddleware', - 'django.contrib.auth.middleware.AuthenticationMiddleware', - 'django.contrib.messages.middleware.MessageMiddleware', - 'django.middleware.clickjacking.XFrameOptionsMiddleware', -] - -ROOT_URLCONF = 'etesync_server.urls' - -TEMPLATES = [ - { - 'BACKEND': 'django.template.backends.django.DjangoTemplates', - 'DIRS': [ - os.path.join(BASE_DIR, 'templates') - ], - 'APP_DIRS': True, - 'OPTIONS': { - 'context_processors': [ - 'django.template.context_processors.debug', - 'django.template.context_processors.request', - 'django.contrib.auth.context_processors.auth', - 'django.contrib.messages.context_processors.messages', - ], - }, - }, -] - -WSGI_APPLICATION = 'etesync_server.wsgi.application' - - -# Password validation -# https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators - -AUTH_PASSWORD_VALIDATORS = [ - { - 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', - }, -] - - -# Internationalization -# https://docs.djangoproject.com/en/2.0/topics/i18n/ - -LANGUAGE_CODE = 'en-us' - -TIME_ZONE = 'UTC' - -USE_I18N = True - -USE_L10N = True - -USE_TZ = True - - -# Static files (CSS, JavaScript, Images) -# https://docs.djangoproject.com/en/2.0/howto/static-files/ - -STATIC_URL = '/static/' -STATIC_ROOT = os.path.join(BASE_DIR, 'static/') - - -# Define where to find configuration files -config_locations = ['etesync-server.ini', '/etc/etesync-server/etesync-server.ini'] -# Use config file if present -if any(os.path.isfile(x) for x in config_locations): - config = configparser.ConfigParser() - config.read(config_locations) - - section = config['global'] - - SECRET_FILE = section.get('secret_file', SECRET_FILE) - STATIC_ROOT = section.get('static_root', STATIC_ROOT) - STATIC_URL = section.get('static_url', STATIC_URL) - LANGUAGE_CODE = section.get('language_code', LANGUAGE_CODE) - TIME_ZONE = section.get('time_zone', TIME_ZONE) - DEBUG = section.getboolean('debug', DEBUG) - - if 'allowed_hosts' in config: - ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')] - - if 'database' in config: - DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } } - -JOURNAL_API_PERMISSIONS = ( - 'rest_framework.permissions.IsAuthenticated', - ) - -# Cors -CORS_ORIGIN_ALLOW_ALL = True - -# Make an `etesync_site_settings` module available to override settings. -try: - from etesync_site_settings import * -except ImportError: - pass - -if 'SECRET_KEY' not in locals(): - SECRET_KEY = get_secret_from_file(SECRET_FILE) diff --git a/etesync_server/urls.py b/etesync_server/urls.py deleted file mode 100644 index feb8892..0000000 --- a/etesync_server/urls.py +++ /dev/null @@ -1,70 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -"""etesync_server URL Configuration - -The `urlpatterns` list routes URLs to views. For more information please see: - https://docs.djangoproject.com/en/2.0/topics/http/urls/ -Examples: -Function views - 1. Add an import: from my_app import views - 2. Add a URL to urlpatterns: path('', views.home, name='home') -Class-based views - 1. Add an import: from other_app.views import Home - 2. Add a URL to urlpatterns: path('', Home.as_view(), name='home') -Including another URLconf - 1. Import the include() function: from django.urls import include, path - 2. Add a URL to urlpatterns: path('blog/', include('blog.urls')) -""" -from django.conf import settings -from django.urls import include, path, re_path -from django.contrib import admin -from django.views.generic import TemplateView - -from rest_framework_nested import routers -from rest_framework.authtoken import views as token_views -from rest_framework.decorators import api_view -from rest_framework.response import Response - -from journal import views - -router = routers.DefaultRouter() -router.register(r'journals', views.JournalViewSet) -router.register(r'journal/(?P[^/]+)', views.EntryViewSet) -router.register(r'user', views.UserInfoViewSet) - -journals_router = routers.NestedSimpleRouter(router, r'journals', lookup='journal') -journals_router.register(r'members', views.MembersViewSet, basename='journal-members') -journals_router.register(r'entries', views.EntryViewSet, basename='journal-entries') - - -@api_view(['POST']) -def nop_view(request): - return Response({}) - - -urlpatterns = [ - re_path(r'^api/v1/', include(router.urls)), - re_path(r'^api/v1/', include(journals_router.urls)), - re_path(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), - re_path(r'^api-token-auth/', token_views.obtain_auth_token), - path('api/logout/', nop_view), - path('admin/', admin.site.urls), - path('', TemplateView.as_view(template_name='success.html')), -] - -if settings.DEBUG: - urlpatterns = [ - path('reset/', views.reset, name='reset_debug'), - ] + urlpatterns diff --git a/etesync_server/utils.py b/etesync_server/utils.py deleted file mode 100644 index 21c99f2..0000000 --- a/etesync_server/utils.py +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -from django.core.management import utils - -def get_secret_from_file(path): - try: - with open(path, "r") as f: - return f.read().strip() - except EnvironmentError: - with open(path, "w") as f: - secret_key = utils.get_random_secret_key() - f.write(secret_key) - return secret_key diff --git a/etesync_server/wsgi.py b/etesync_server/wsgi.py deleted file mode 100644 index 6738999..0000000 --- a/etesync_server/wsgi.py +++ /dev/null @@ -1,16 +0,0 @@ -""" -WSGI config for etesync_server project. - -It exposes the WSGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/2.0/howto/deployment/wsgi/ -""" - -import os - -from django.core.wsgi import get_wsgi_application - -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etesync_server.settings") - -application = get_wsgi_application() From 801826b8b69e929c3ec2c9b9e0410ebbc6b6f9b5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 5 Nov 2020 16:50:09 +0200 Subject: [PATCH 315/511] Fix the URL in the success template. --- templates/success.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/templates/success.html b/templates/success.html index c7cf494..c3b696f 100644 --- a/templates/success.html +++ b/templates/success.html @@ -6,7 +6,7 @@

It works!

- Please refer to the README to complete the final steps if you haven't done so already. + Please refer to the README to complete the final steps if you haven't done so already.

From 47103df48a7c9de61f359bdbd0bba4235a1f92be Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 7 Nov 2020 18:58:29 +0200 Subject: [PATCH 316/511] Change user creation to not ask for a password (and clarify the readme). --- README.md | 7 +++---- myauth/admin.py | 13 ++++++++++++- myauth/forms.py | 30 ++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 5 deletions(-) create mode 100644 myauth/forms.py diff --git a/README.md b/README.md index 0ecf52a..3a3f5fd 100644 --- a/README.md +++ b/README.md @@ -84,10 +84,9 @@ Create yourself an admin user: ``` At this stage you need to create accounts to be used with the EteSync apps. To do that, please go to: -`www.your-etesync-install.com/admin` and create a new user to be used with the service. Set a random -password for the user such as `j3PmCRftyQMtM3eWvi8f`. No need to remember it, as it won't be used. -Etebase uses a zero-knowledge proof for authentication, so the user will just create a password when -creating the account from the apps. +`www.your-etesync-install.com/admin` and create a new user to be used with the service. No need to set +a password, as Etebase uses a zero-knowledge proof for authentication, so the user will just create +a password when creating the account from the apps. After this user has been created, you can use any of the EteSync apps to signup (or login) with the same username and email in order to set up the account. The password used at that point will be used to setup the account. diff --git a/myauth/admin.py b/myauth/admin.py index f91be8f..0ecde3f 100644 --- a/myauth/admin.py +++ b/myauth/admin.py @@ -1,5 +1,16 @@ from django.contrib import admin -from django.contrib.auth.admin import UserAdmin +from django.contrib.auth.admin import UserAdmin as DjangoUserAdmin from .models import User +from .forms import AdminUserCreationForm + + +class UserAdmin(DjangoUserAdmin): + add_form = AdminUserCreationForm + add_fieldsets = ( + (None, { + 'classes': ('wide',), + 'fields': ('username', ), + }), + ) admin.site.register(User, UserAdmin) diff --git a/myauth/forms.py b/myauth/forms.py new file mode 100644 index 0000000..55f7299 --- /dev/null +++ b/myauth/forms.py @@ -0,0 +1,30 @@ +from django import forms +from django.contrib.auth import get_user_model +from django.contrib.auth.forms import UsernameField + +User = get_user_model() + + +class AdminUserCreationForm(forms.ModelForm): + """ + A form that creates a user, with no privileges, from the given username and + password. + """ + + class Meta: + model = User + fields = ("username",) + field_classes = {'username': UsernameField} + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + if self._meta.model.USERNAME_FIELD in self.fields: + self.fields[self._meta.model.USERNAME_FIELD].widget.attrs['autofocus'] = True + + def save(self, commit=True): + user = super().save(commit=False) + user.set_unusable_password() + if commit: + user.save() + return user + From e9de8f1adb16b47becf700100333c38b8e860f5c Mon Sep 17 00:00:00 2001 From: "Victor R. Santos" Date: Sat, 7 Nov 2020 16:21:34 -0300 Subject: [PATCH 317/511] Add env variable to change configuration file path. ETEBASE_EASY_CONFIG_PATH is optional, the server serches for the configurations files in this order: - "ETEBASE_EASY_CONFIG_PATH" - etebase-server.ini - /etc/etebase-server/etebase-server.ini --- etebase_server/settings.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/etebase_server/settings.py b/etebase_server/settings.py index f785cb7..ee98f55 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -141,7 +141,12 @@ MEDIA_URL = '/user-media/' # Define where to find configuration files -config_locations = ['etebase-server.ini', '/etc/etebase-server/etebase-server.ini'] +config_locations = [ + os.environ.get('ETEBASE_EASY_CONFIG_PATH', ''), + 'etebase-server.ini', + '/etc/etebase-server/etebase-server.ini', +] + # Use config file if present if any(os.path.isfile(x) for x in config_locations): config = configparser.ConfigParser() From bdd787b9158cee532fc9ba1e579d7ef7337dcf8a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 9 Nov 2020 17:31:12 +0200 Subject: [PATCH 318/511] Gracefully handle uploading the same item twice. We were failing until now, but since the uid is sure to be unique, we can just assume that if it's the same uid it's the same content. This means we can just gracefully fail as the data is the same. Until now, we were raising an error, but we now just do nothing and consider it a success. This is especially useful when a network error caused an item to be uploaded but not updated on the client side. --- django_etebase/serializers.py | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 97dcd64..4f2d802 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -30,6 +30,10 @@ User = get_user_model() def process_revisions_for_item(item, revision_data): chunks_objs = [] chunks = revision_data.pop('chunks_relation') + + revision = models.CollectionItemRevision(**revision_data, item=item) + revision.validate_unique() # Verify there aren't any validation issues + for chunk in chunks: uid = chunk[0] chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() @@ -47,8 +51,9 @@ def process_revisions_for_item(item, revision_data): chunks_objs.append(chunk_obj) stoken = models.Stoken.objects.create() + revision.stoken = stoken + revision.save() - revision = models.CollectionItemRevision.objects.create(**revision_data, item=item, stoken=stoken) for chunk in chunks_objs: models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) return revision @@ -196,6 +201,9 @@ class CollectionItemRevisionSerializer(BetterErrorsMixin, serializers.ModelSeria class Meta: model = models.CollectionItemRevision fields = ('chunks', 'meta', 'uid', 'deleted') + extra_kwargs = { + 'uid': {'validators': []}, # We deal with it in the serializers + } class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): @@ -220,6 +228,10 @@ class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): instance, created = Model.objects.get_or_create(uid=uid, defaults=validated_data) cur_etag = instance.etag if not created else None + # If we are trying to update an up to date item, abort early and consider it a success + if cur_etag == revision_data.get('uid'): + return instance + if validate_etag and cur_etag != etag: raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(cur_etag, etag), status_code=status.HTTP_409_CONFLICT) @@ -231,7 +243,10 @@ class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): current_revision.current = None current_revision.save() - process_revisions_for_item(instance, revision_data) + try: + process_revisions_for_item(instance, revision_data) + except django_exceptions.ValidationError as e: + self.transform_validation_error("content", e) return instance From ab8b2bc58aba87e40d8df91af83dd4e9c1519b62 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 12 Nov 2020 14:07:27 +0200 Subject: [PATCH 319/511] README: update + add chat badge. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3a3f5fd..ebb384e 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,9 @@

Etebase - Encrypt Everything

-A skeleton app for running your own [Etebase](https://www.etebase.com) (EteSync 2.0) server. +An [Etebase](https://www.etebase.com) (EteSync 2.0) server so you can run your own. + +[![Chat with us](https://img.shields.io/badge/chat-IRC%20|%20Matrix%20|%20Web-blue.svg)](https://www.etebase.com/community-chat/) # Installation From 9ec16e921623acc80a405aaf96306e0e9ece83fe Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sat, 14 Nov 2020 16:56:26 +0200 Subject: [PATCH 320/511] Update changelog. --- ChangeLog.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index 8845089..d3adef0 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,10 @@ # Changelog +## Version 0.5.2 +* Fix issues with host verification failing with a custom port +* Add env variable to change configuration file path. +* Change user creation to not ask for a password (and clarify the readme). + ## Version 0.5.1 * Enforce collections to always have a collection type set * Collection saving: add another verification for collection UID uniqueness. From d8e5c37db1b5876ca0664721647b7c14bf4d8a9e Mon Sep 17 00:00:00 2001 From: Tal Leibman <42600279+Tal-Leibman@users.noreply.github.com> Date: Sat, 14 Nov 2020 17:04:41 +0200 Subject: [PATCH 321/511] Use black for code formatting and format the code Merge #65 --- django_etebase/app_settings.py | 16 +- django_etebase/apps.py | 2 +- django_etebase/drf_msgpack/apps.py | 2 +- django_etebase/drf_msgpack/parsers.py | 4 +- django_etebase/drf_msgpack/renderers.py | 8 +- django_etebase/exceptions.py | 7 +- django_etebase/migrations/0001_initial.py | 180 +++++--- django_etebase/migrations/0002_userinfo.py | 22 +- .../migrations/0003_collectioninvitation.py | 50 ++- .../0004_collectioninvitation_version.py | 6 +- .../migrations/0005_auto_20200526_1021.py | 8 +- .../migrations/0006_auto_20200526_1040.py | 14 +- .../migrations/0007_auto_20200526_1336.py | 66 ++- .../migrations/0008_auto_20200526_1535.py | 29 +- .../migrations/0009_auto_20200526_1535.py | 6 +- .../migrations/0010_auto_20200526_1539.py | 8 +- .../0011_collectionmember_stoken.py | 10 +- .../migrations/0012_auto_20200527_0743.py | 6 +- .../0013_collectionmemberremoved.py | 28 +- .../migrations/0014_auto_20200602_1558.py | 8 +- .../0015_collectionitemrevision_salt.py | 6 +- .../migrations/0016_auto_20200623_0820.py | 28 +- .../migrations/0017_auto_20200623_0958.py | 23 +- .../migrations/0018_auto_20200624_0748.py | 14 +- .../migrations/0019_auto_20200626_0748.py | 14 +- ...0020_remove_collectionitemrevision_salt.py | 7 +- .../migrations/0021_auto_20200626_0913.py | 65 ++- .../migrations/0022_auto_20200804_1059.py | 7 +- .../0023_collectionitemchunk_collection.py | 13 +- .../migrations/0024_auto_20200804_1209.py | 4 +- .../migrations/0025_auto_20200804_1216.py | 20 +- .../migrations/0026_auto_20200907_0752.py | 14 +- .../migrations/0027_auto_20200907_0752.py | 14 +- .../migrations/0028_auto_20200907_0754.py | 18 +- .../migrations/0029_auto_20200907_0801.py | 12 +- .../migrations/0030_auto_20200922_0832.py | 13 +- .../migrations/0031_auto_20201013_1336.py | 18 +- .../migrations/0032_auto_20201013_1409.py | 6 +- django_etebase/models.py | 114 ++--- django_etebase/parsers.py | 7 +- django_etebase/permissions.py | 21 +- django_etebase/renderers.py | 1 + django_etebase/serializers.py | 237 ++++++----- django_etebase/signals.py | 2 +- django_etebase/token_auth/apps.py | 2 +- django_etebase/token_auth/authentication.py | 10 +- .../token_auth/migrations/0001_initial.py | 22 +- django_etebase/token_auth/models.py | 5 +- django_etebase/urls.py | 28 +- django_etebase/utils.py | 4 +- django_etebase/views.py | 395 +++++++++--------- etebase_server/asgi.py | 2 +- etebase_server/settings.py | 143 +++---- etebase_server/urls.py | 9 +- etebase_server/utils.py | 1 + etebase_server/wsgi.py | 2 +- manage.py | 4 +- myauth/admin.py | 8 +- myauth/apps.py | 2 +- myauth/forms.py | 5 +- myauth/migrations/0001_initial.py | 91 +++- myauth/migrations/0002_auto_20200515_0801.py | 15 +- myauth/models.py | 17 +- pyproject.toml | 2 + requirements.in/development.txt | 1 + 65 files changed, 1094 insertions(+), 832 deletions(-) create mode 100644 pyproject.toml diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index 3c580b2..7c93f5f 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -21,18 +21,19 @@ class AppSettings: def import_from_str(self, name): from importlib import import_module - path, prop = name.rsplit('.', 1) + path, prop = name.rsplit(".", 1) mod = import_module(path) return getattr(mod, prop) def _setting(self, name, dflt): from django.conf import settings + return getattr(settings, self.prefix + name, dflt) @cached_property def API_PERMISSIONS(self): # pylint: disable=invalid-name - perms = self._setting("API_PERMISSIONS", ('rest_framework.permissions.IsAuthenticated', )) + perms = self._setting("API_PERMISSIONS", ("rest_framework.permissions.IsAuthenticated",)) ret = [] for perm in perms: ret.append(self.import_from_str(perm)) @@ -40,8 +41,13 @@ class AppSettings: @cached_property def API_AUTHENTICATORS(self): # pylint: disable=invalid-name - perms = self._setting("API_AUTHENTICATORS", ('rest_framework.authentication.TokenAuthentication', - 'rest_framework.authentication.SessionAuthentication')) + perms = self._setting( + "API_AUTHENTICATORS", + ( + "rest_framework.authentication.TokenAuthentication", + "rest_framework.authentication.SessionAuthentication", + ), + ) ret = [] for perm in perms: ret.append(self.import_from_str(perm)) @@ -80,4 +86,4 @@ class AppSettings: return self._setting("CHALLENGE_VALID_SECONDS", 60) -app_settings = AppSettings('ETEBASE_') +app_settings = AppSettings("ETEBASE_") diff --git a/django_etebase/apps.py b/django_etebase/apps.py index 286a708..84e4b6e 100644 --- a/django_etebase/apps.py +++ b/django_etebase/apps.py @@ -2,4 +2,4 @@ from django.apps import AppConfig class DjangoEtebaseConfig(AppConfig): - name = 'django_etebase' + name = "django_etebase" diff --git a/django_etebase/drf_msgpack/apps.py b/django_etebase/drf_msgpack/apps.py index 619e3e0..22ea2c1 100644 --- a/django_etebase/drf_msgpack/apps.py +++ b/django_etebase/drf_msgpack/apps.py @@ -2,4 +2,4 @@ from django.apps import AppConfig class DrfMsgpackConfig(AppConfig): - name = 'drf_msgpack' + name = "drf_msgpack" diff --git a/django_etebase/drf_msgpack/parsers.py b/django_etebase/drf_msgpack/parsers.py index 44cd33b..0504a76 100644 --- a/django_etebase/drf_msgpack/parsers.py +++ b/django_etebase/drf_msgpack/parsers.py @@ -5,10 +5,10 @@ from rest_framework.exceptions import ParseError class MessagePackParser(BaseParser): - media_type = 'application/msgpack' + media_type = "application/msgpack" def parse(self, stream, media_type=None, parser_context=None): try: return msgpack.unpackb(stream.read(), raw=False) except Exception as exc: - raise ParseError('MessagePack parse error - %s' % str(exc)) + raise ParseError("MessagePack parse error - %s" % str(exc)) diff --git a/django_etebase/drf_msgpack/renderers.py b/django_etebase/drf_msgpack/renderers.py index 9445231..35a4afa 100644 --- a/django_etebase/drf_msgpack/renderers.py +++ b/django_etebase/drf_msgpack/renderers.py @@ -4,12 +4,12 @@ from rest_framework.renderers import BaseRenderer class MessagePackRenderer(BaseRenderer): - media_type = 'application/msgpack' - format = 'msgpack' - render_style = 'binary' + media_type = "application/msgpack" + format = "msgpack" + render_style = "binary" charset = None def render(self, data, media_type=None, renderer_context=None): if data is None: - return b'' + return b"" return msgpack.packb(data, use_bin_type=True) diff --git a/django_etebase/exceptions.py b/django_etebase/exceptions.py index d05c4e5..f3aa08a 100644 --- a/django_etebase/exceptions.py +++ b/django_etebase/exceptions.py @@ -3,8 +3,7 @@ from rest_framework import serializers, status class EtebaseValidationError(serializers.ValidationError): def __init__(self, code, detail, status_code=status.HTTP_400_BAD_REQUEST): - super().__init__({ - 'code': code, - 'detail': detail, - }) + super().__init__( + {"code": code, "detail": detail,} + ) self.status_code = status_code diff --git a/django_etebase/migrations/0001_initial.py b/django_etebase/migrations/0001_initial.py index 69a9a91..86f0fa6 100644 --- a/django_etebase/migrations/0001_initial.py +++ b/django_etebase/migrations/0001_initial.py @@ -17,75 +17,159 @@ class Migration(migrations.Migration): operations = [ migrations.CreateModel( - name='Collection', + name="Collection", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')])), - ('version', models.PositiveSmallIntegerField()), - ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "uid", + models.CharField( + db_index=True, + max_length=44, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="[a-zA-Z0-9]") + ], + ), + ), + ("version", models.PositiveSmallIntegerField()), + ("owner", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={ - 'unique_together': {('uid', 'owner')}, - }, + options={"unique_together": {("uid", "owner")},}, ), migrations.CreateModel( - name='CollectionItem', + name="CollectionItem", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='[a-zA-Z0-9]')])), - ('version', models.PositiveSmallIntegerField()), - ('encryptionKey', models.BinaryField(editable=True, null=True)), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='items', to='django_etebase.Collection')), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "uid", + models.CharField( + db_index=True, + max_length=44, + null=True, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="[a-zA-Z0-9]") + ], + ), + ), + ("version", models.PositiveSmallIntegerField()), + ("encryptionKey", models.BinaryField(editable=True, null=True)), + ( + "collection", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="items", + to="django_etebase.Collection", + ), + ), ], - options={ - 'unique_together': {('uid', 'collection')}, - }, + options={"unique_together": {("uid", "collection")},}, ), migrations.CreateModel( - name='CollectionItemChunk', + name="CollectionItemChunk", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), - ('chunkFile', models.FileField(max_length=150, unique=True, upload_to=django_etebase.models.chunk_directory_path)), - ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etebase.CollectionItem')), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "uid", + models.CharField( + db_index=True, + max_length=44, + validators=[ + django.core.validators.RegexValidator( + message="Expected a 256bit base64url.", regex="^[a-zA-Z0-9\\-_]{43}$" + ) + ], + ), + ), + ( + "chunkFile", + models.FileField(max_length=150, unique=True, upload_to=django_etebase.models.chunk_directory_path), + ), + ( + "item", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="chunks", + to="django_etebase.CollectionItem", + ), + ), ], ), migrations.CreateModel( - name='CollectionItemRevision', + name="CollectionItemRevision", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), - ('meta', models.BinaryField(editable=True)), - ('current', models.BooleanField(db_index=True, default=True, null=True)), - ('deleted', models.BooleanField(default=False)), - ('item', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions', to='django_etebase.CollectionItem')), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "uid", + models.CharField( + db_index=True, + max_length=44, + unique=True, + validators=[ + django.core.validators.RegexValidator( + message="Expected a 256bit base64url.", regex="^[a-zA-Z0-9\\-_]{43}$" + ) + ], + ), + ), + ("meta", models.BinaryField(editable=True)), + ("current", models.BooleanField(db_index=True, default=True, null=True)), + ("deleted", models.BooleanField(default=False)), + ( + "item", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="revisions", + to="django_etebase.CollectionItem", + ), + ), ], - options={ - 'unique_together': {('item', 'current')}, - }, + options={"unique_together": {("item", "current")},}, ), migrations.CreateModel( - name='RevisionChunkRelation', + name="RevisionChunkRelation", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('chunk', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='revisions_relation', to='django_etebase.CollectionItemChunk')), - ('revision', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks_relation', to='django_etebase.CollectionItemRevision')), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "chunk", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="revisions_relation", + to="django_etebase.CollectionItemChunk", + ), + ), + ( + "revision", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="chunks_relation", + to="django_etebase.CollectionItemRevision", + ), + ), ], - options={ - 'ordering': ('id',), - }, + options={"ordering": ("id",),}, ), migrations.CreateModel( - name='CollectionMember', + name="CollectionMember", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('encryptionKey', models.BinaryField(editable=True)), - ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='members', to='django_etebase.Collection')), - ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ("encryptionKey", models.BinaryField(editable=True)), + ( + "accessLevel", + models.CharField( + choices=[("adm", "Admin"), ("rw", "Read Write"), ("ro", "Read Only")], + default="ro", + max_length=3, + ), + ), + ( + "collection", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="members", + to="django_etebase.Collection", + ), + ), + ("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={ - 'unique_together': {('user', 'collection')}, - }, + options={"unique_together": {("user", "collection")},}, ), ] diff --git a/django_etebase/migrations/0002_userinfo.py b/django_etebase/migrations/0002_userinfo.py index 6da0bb8..6ddd9a5 100644 --- a/django_etebase/migrations/0002_userinfo.py +++ b/django_etebase/migrations/0002_userinfo.py @@ -8,18 +8,26 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('myauth', '0001_initial'), - ('django_etebase', '0001_initial'), + ("myauth", "0001_initial"), + ("django_etebase", "0001_initial"), ] operations = [ migrations.CreateModel( - name='UserInfo', + name="UserInfo", fields=[ - ('owner', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, primary_key=True, serialize=False, to=settings.AUTH_USER_MODEL)), - ('version', models.PositiveSmallIntegerField(default=1)), - ('pubkey', models.BinaryField(editable=True)), - ('salt', models.BinaryField(editable=True)), + ( + "owner", + models.OneToOneField( + on_delete=django.db.models.deletion.CASCADE, + primary_key=True, + serialize=False, + to=settings.AUTH_USER_MODEL, + ), + ), + ("version", models.PositiveSmallIntegerField(default=1)), + ("pubkey", models.BinaryField(editable=True)), + ("salt", models.BinaryField(editable=True)), ], ), ] diff --git a/django_etebase/migrations/0003_collectioninvitation.py b/django_etebase/migrations/0003_collectioninvitation.py index 8fd2066..1b416ab 100644 --- a/django_etebase/migrations/0003_collectioninvitation.py +++ b/django_etebase/migrations/0003_collectioninvitation.py @@ -10,22 +10,50 @@ class Migration(migrations.Migration): dependencies = [ migrations.swappable_dependency(settings.AUTH_USER_MODEL), - ('django_etebase', '0002_userinfo'), + ("django_etebase", "0002_userinfo"), ] operations = [ migrations.CreateModel( - name='CollectionInvitation', + name="CollectionInvitation", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, max_length=44, validators=[django.core.validators.RegexValidator(message='Expected a 256bit base64url.', regex='^[a-zA-Z0-9\\-_]{43}$')])), - ('signedEncryptionKey', models.BinaryField()), - ('accessLevel', models.CharField(choices=[('adm', 'Admin'), ('rw', 'Read Write'), ('ro', 'Read Only')], default='ro', max_length=3)), - ('fromMember', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='django_etebase.CollectionMember')), - ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='incoming_invitations', to=settings.AUTH_USER_MODEL)), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "uid", + models.CharField( + db_index=True, + max_length=44, + validators=[ + django.core.validators.RegexValidator( + message="Expected a 256bit base64url.", regex="^[a-zA-Z0-9\\-_]{43}$" + ) + ], + ), + ), + ("signedEncryptionKey", models.BinaryField()), + ( + "accessLevel", + models.CharField( + choices=[("adm", "Admin"), ("rw", "Read Write"), ("ro", "Read Only")], + default="ro", + max_length=3, + ), + ), + ( + "fromMember", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, to="django_etebase.CollectionMember" + ), + ), + ( + "user", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="incoming_invitations", + to=settings.AUTH_USER_MODEL, + ), + ), ], - options={ - 'unique_together': {('user', 'fromMember')}, - }, + options={"unique_together": {("user", "fromMember")},}, ), ] diff --git a/django_etebase/migrations/0004_collectioninvitation_version.py b/django_etebase/migrations/0004_collectioninvitation_version.py index 4052116..29ae3f1 100644 --- a/django_etebase/migrations/0004_collectioninvitation_version.py +++ b/django_etebase/migrations/0004_collectioninvitation_version.py @@ -6,13 +6,11 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0003_collectioninvitation'), + ("django_etebase", "0003_collectioninvitation"), ] operations = [ migrations.AddField( - model_name='collectioninvitation', - name='version', - field=models.PositiveSmallIntegerField(default=1), + model_name="collectioninvitation", name="version", field=models.PositiveSmallIntegerField(default=1), ), ] diff --git a/django_etebase/migrations/0005_auto_20200526_1021.py b/django_etebase/migrations/0005_auto_20200526_1021.py index da0dc33..3775277 100644 --- a/django_etebase/migrations/0005_auto_20200526_1021.py +++ b/django_etebase/migrations/0005_auto_20200526_1021.py @@ -6,13 +6,9 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0004_collectioninvitation_version'), + ("django_etebase", "0004_collectioninvitation_version"), ] operations = [ - migrations.RenameField( - model_name='userinfo', - old_name='pubkey', - new_name='loginPubkey', - ), + migrations.RenameField(model_name="userinfo", old_name="pubkey", new_name="loginPubkey",), ] diff --git a/django_etebase/migrations/0006_auto_20200526_1040.py b/django_etebase/migrations/0006_auto_20200526_1040.py index b86a996..07b01cd 100644 --- a/django_etebase/migrations/0006_auto_20200526_1040.py +++ b/django_etebase/migrations/0006_auto_20200526_1040.py @@ -6,20 +6,20 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0005_auto_20200526_1021'), + ("django_etebase", "0005_auto_20200526_1021"), ] operations = [ migrations.AddField( - model_name='userinfo', - name='encryptedSeckey', - field=models.BinaryField(default=b'', editable=True), + model_name="userinfo", + name="encryptedSeckey", + field=models.BinaryField(default=b"", editable=True), preserve_default=False, ), migrations.AddField( - model_name='userinfo', - name='pubkey', - field=models.BinaryField(default=b'', editable=True), + model_name="userinfo", + name="pubkey", + field=models.BinaryField(default=b"", editable=True), preserve_default=False, ), ] diff --git a/django_etebase/migrations/0007_auto_20200526_1336.py b/django_etebase/migrations/0007_auto_20200526_1336.py index 79978c7..01afe45 100644 --- a/django_etebase/migrations/0007_auto_20200526_1336.py +++ b/django_etebase/migrations/0007_auto_20200526_1336.py @@ -7,33 +7,67 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0006_auto_20200526_1040'), + ("django_etebase", "0006_auto_20200526_1040"), ] operations = [ migrations.AlterField( - model_name='collection', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9]*$')]), + model_name="collection", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9]*$")], + ), ), migrations.AlterField( - model_name='collectioninvitation', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')]), + model_name="collectioninvitation", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[ + django.core.validators.RegexValidator( + message="Expected a base64url.", regex="^[a-zA-Z0-9\\-_]{42,43}$" + ) + ], + ), ), migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=43, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9]*$')]), + model_name="collectionitem", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + null=True, + validators=[django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9]*$")], + ), ), migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')]), + model_name="collectionitemchunk", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[ + django.core.validators.RegexValidator( + message="Expected a base64url.", regex="^[a-zA-Z0-9\\-_]{42,43}$" + ) + ], + ), ), migrations.AlterField( - model_name='collectionitemrevision', - name='uid', - field=models.CharField(db_index=True, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')]), + model_name="collectionitemrevision", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + unique=True, + validators=[ + django.core.validators.RegexValidator( + message="Expected a base64url.", regex="^[a-zA-Z0-9\\-_]{42,43}$" + ) + ], + ), ), ] diff --git a/django_etebase/migrations/0008_auto_20200526_1535.py b/django_etebase/migrations/0008_auto_20200526_1535.py index 12656c0..7bb83d5 100644 --- a/django_etebase/migrations/0008_auto_20200526_1535.py +++ b/django_etebase/migrations/0008_auto_20200526_1535.py @@ -9,20 +9,35 @@ import django_etebase.models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0007_auto_20200526_1336'), + ("django_etebase", "0007_auto_20200526_1336"), ] operations = [ migrations.CreateModel( - name='Stoken', + name="Stoken", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.CharField(db_index=True, default=django_etebase.models.generate_stoken_uid, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Expected a base64url.', regex='^[a-zA-Z0-9\\-_]{42,43}$')])), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "uid", + models.CharField( + db_index=True, + default=django_etebase.models.generate_stoken_uid, + max_length=43, + unique=True, + validators=[ + django.core.validators.RegexValidator( + message="Expected a base64url.", regex="^[a-zA-Z0-9\\-_]{42,43}$" + ) + ], + ), + ), ], ), migrations.AddField( - model_name='collectionitemrevision', - name='stoken', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken'), + model_name="collectionitemrevision", + name="stoken", + field=models.OneToOneField( + null=True, on_delete=django.db.models.deletion.PROTECT, to="django_etebase.Stoken" + ), ), ] diff --git a/django_etebase/migrations/0009_auto_20200526_1535.py b/django_etebase/migrations/0009_auto_20200526_1535.py index a6ff498..0ab2f8c 100644 --- a/django_etebase/migrations/0009_auto_20200526_1535.py +++ b/django_etebase/migrations/0009_auto_20200526_1535.py @@ -4,8 +4,8 @@ from django.db import migrations def create_stokens(apps, schema_editor): - Stoken = apps.get_model('django_etebase', 'Stoken') - CollectionItemRevision = apps.get_model('django_etebase', 'CollectionItemRevision') + Stoken = apps.get_model("django_etebase", "Stoken") + CollectionItemRevision = apps.get_model("django_etebase", "CollectionItemRevision") for rev in CollectionItemRevision.objects.all(): rev.stoken = Stoken.objects.create() @@ -15,7 +15,7 @@ def create_stokens(apps, schema_editor): class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0008_auto_20200526_1535'), + ("django_etebase", "0008_auto_20200526_1535"), ] operations = [ diff --git a/django_etebase/migrations/0010_auto_20200526_1539.py b/django_etebase/migrations/0010_auto_20200526_1539.py index 7ef0eca..204b97d 100644 --- a/django_etebase/migrations/0010_auto_20200526_1539.py +++ b/django_etebase/migrations/0010_auto_20200526_1539.py @@ -7,13 +7,13 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0009_auto_20200526_1535'), + ("django_etebase", "0009_auto_20200526_1535"), ] operations = [ migrations.AlterField( - model_name='collectionitemrevision', - name='stoken', - field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken'), + model_name="collectionitemrevision", + name="stoken", + field=models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, to="django_etebase.Stoken"), ), ] diff --git a/django_etebase/migrations/0011_collectionmember_stoken.py b/django_etebase/migrations/0011_collectionmember_stoken.py index bafaea7..cbe8d06 100644 --- a/django_etebase/migrations/0011_collectionmember_stoken.py +++ b/django_etebase/migrations/0011_collectionmember_stoken.py @@ -7,13 +7,15 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0010_auto_20200526_1539'), + ("django_etebase", "0010_auto_20200526_1539"), ] operations = [ migrations.AddField( - model_name='collectionmember', - name='stoken', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken'), + model_name="collectionmember", + name="stoken", + field=models.OneToOneField( + null=True, on_delete=django.db.models.deletion.PROTECT, to="django_etebase.Stoken" + ), ), ] diff --git a/django_etebase/migrations/0012_auto_20200527_0743.py b/django_etebase/migrations/0012_auto_20200527_0743.py index ab6adbc..1f58f82 100644 --- a/django_etebase/migrations/0012_auto_20200527_0743.py +++ b/django_etebase/migrations/0012_auto_20200527_0743.py @@ -4,8 +4,8 @@ from django.db import migrations def create_stokens(apps, schema_editor): - Stoken = apps.get_model('django_etebase', 'Stoken') - CollectionMember = apps.get_model('django_etebase', 'CollectionMember') + Stoken = apps.get_model("django_etebase", "Stoken") + CollectionMember = apps.get_model("django_etebase", "CollectionMember") for member in CollectionMember.objects.all(): member.stoken = Stoken.objects.create() @@ -15,7 +15,7 @@ def create_stokens(apps, schema_editor): class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0011_collectionmember_stoken'), + ("django_etebase", "0011_collectionmember_stoken"), ] operations = [ diff --git a/django_etebase/migrations/0013_collectionmemberremoved.py b/django_etebase/migrations/0013_collectionmemberremoved.py index 2641c03..4481e80 100644 --- a/django_etebase/migrations/0013_collectionmemberremoved.py +++ b/django_etebase/migrations/0013_collectionmemberremoved.py @@ -9,20 +9,30 @@ class Migration(migrations.Migration): dependencies = [ migrations.swappable_dependency(settings.AUTH_USER_MODEL), - ('django_etebase', '0012_auto_20200527_0743'), + ("django_etebase", "0012_auto_20200527_0743"), ] operations = [ migrations.CreateModel( - name='CollectionMemberRemoved', + name="CollectionMemberRemoved", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='removed_members', to='django_etebase.Collection')), - ('stoken', models.OneToOneField(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.Stoken')), - ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "collection", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="removed_members", + to="django_etebase.Collection", + ), + ), + ( + "stoken", + models.OneToOneField( + null=True, on_delete=django.db.models.deletion.PROTECT, to="django_etebase.Stoken" + ), + ), + ("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={ - 'unique_together': {('user', 'collection')}, - }, + options={"unique_together": {("user", "collection")},}, ), ] diff --git a/django_etebase/migrations/0014_auto_20200602_1558.py b/django_etebase/migrations/0014_auto_20200602_1558.py index d1a555d..42bed52 100644 --- a/django_etebase/migrations/0014_auto_20200602_1558.py +++ b/django_etebase/migrations/0014_auto_20200602_1558.py @@ -6,13 +6,9 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0013_collectionmemberremoved'), + ("django_etebase", "0013_collectionmemberremoved"), ] operations = [ - migrations.RenameField( - model_name='userinfo', - old_name='encryptedSeckey', - new_name='encryptedContent', - ), + migrations.RenameField(model_name="userinfo", old_name="encryptedSeckey", new_name="encryptedContent",), ] diff --git a/django_etebase/migrations/0015_collectionitemrevision_salt.py b/django_etebase/migrations/0015_collectionitemrevision_salt.py index 7f3dd71..c4dc3e9 100644 --- a/django_etebase/migrations/0015_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0015_collectionitemrevision_salt.py @@ -6,13 +6,11 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0014_auto_20200602_1558'), + ("django_etebase", "0014_auto_20200602_1558"), ] operations = [ migrations.AddField( - model_name='collectionitemrevision', - name='salt', - field=models.BinaryField(default=b'', editable=True), + model_name="collectionitemrevision", name="salt", field=models.BinaryField(default=b"", editable=True), ), ] diff --git a/django_etebase/migrations/0016_auto_20200623_0820.py b/django_etebase/migrations/0016_auto_20200623_0820.py index 2c11157..a273b0d 100644 --- a/django_etebase/migrations/0016_auto_20200623_0820.py +++ b/django_etebase/migrations/0016_auto_20200623_0820.py @@ -7,25 +7,21 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0015_collectionitemrevision_salt'), + ("django_etebase", "0015_collectionitemrevision_salt"), ] operations = [ migrations.AddField( - model_name='collection', - name='main_item', - field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.PROTECT, related_name='parent', to='django_etebase.CollectionItem'), - ), - migrations.AlterUniqueTogether( - name='collection', - unique_together=set(), - ), - migrations.RemoveField( - model_name='collection', - name='uid', - ), - migrations.RemoveField( - model_name='collection', - name='version', + model_name="collection", + name="main_item", + field=models.ForeignKey( + null=True, + on_delete=django.db.models.deletion.PROTECT, + related_name="parent", + to="django_etebase.CollectionItem", + ), ), + migrations.AlterUniqueTogether(name="collection", unique_together=set(),), + migrations.RemoveField(model_name="collection", name="uid",), + migrations.RemoveField(model_name="collection", name="version",), ] diff --git a/django_etebase/migrations/0017_auto_20200623_0958.py b/django_etebase/migrations/0017_auto_20200623_0958.py index e244b13..dc599aa 100644 --- a/django_etebase/migrations/0017_auto_20200623_0958.py +++ b/django_etebase/migrations/0017_auto_20200623_0958.py @@ -8,18 +8,27 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0016_auto_20200623_0820'), + ("django_etebase", "0016_auto_20200623_0820"), ] operations = [ migrations.AlterField( - model_name='collection', - name='main_item', - field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='parent', to='django_etebase.CollectionItem'), + model_name="collection", + name="main_item", + field=models.ForeignKey( + null=True, + on_delete=django.db.models.deletion.SET_NULL, + related_name="parent", + to="django_etebase.CollectionItem", + ), ), migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9]*$')]), + model_name="collectionitem", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9]*$")], + ), ), ] diff --git a/django_etebase/migrations/0018_auto_20200624_0748.py b/django_etebase/migrations/0018_auto_20200624_0748.py index ec59e0c..d2cdf5a 100644 --- a/django_etebase/migrations/0018_auto_20200624_0748.py +++ b/django_etebase/migrations/0018_auto_20200624_0748.py @@ -7,13 +7,19 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0017_auto_20200623_0958'), + ("django_etebase", "0017_auto_20200623_0958"), ] operations = [ migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]*$')]), + model_name="collectionitem", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]*$") + ], + ), ), ] diff --git a/django_etebase/migrations/0019_auto_20200626_0748.py b/django_etebase/migrations/0019_auto_20200626_0748.py index 991ca50..175e4d0 100644 --- a/django_etebase/migrations/0019_auto_20200626_0748.py +++ b/django_etebase/migrations/0019_auto_20200626_0748.py @@ -7,13 +7,19 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0018_auto_20200624_0748'), + ("django_etebase", "0018_auto_20200624_0748"), ] operations = [ migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=60, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]*$')]), + model_name="collectionitemchunk", + name="uid", + field=models.CharField( + db_index=True, + max_length=60, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]*$") + ], + ), ), ] diff --git a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py index 2df32bf..21d0337 100644 --- a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py @@ -6,12 +6,9 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0019_auto_20200626_0748'), + ("django_etebase", "0019_auto_20200626_0748"), ] operations = [ - migrations.RemoveField( - model_name='collectionitemrevision', - name='salt', - ), + migrations.RemoveField(model_name="collectionitemrevision", name="salt",), ] diff --git a/django_etebase/migrations/0021_auto_20200626_0913.py b/django_etebase/migrations/0021_auto_20200626_0913.py index b890384..3bb6e21 100644 --- a/django_etebase/migrations/0021_auto_20200626_0913.py +++ b/django_etebase/migrations/0021_auto_20200626_0913.py @@ -8,33 +8,66 @@ import django_etebase.models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0020_remove_collectionitemrevision_salt'), + ("django_etebase", "0020_remove_collectionitemrevision_salt"), ] operations = [ migrations.AlterField( - model_name='collectioninvitation', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + model_name="collectioninvitation", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]{20,}$") + ], + ), ), migrations.AlterField( - model_name='collectionitem', - name='uid', - field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + model_name="collectionitem", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]{20,}$") + ], + ), ), migrations.AlterField( - model_name='collectionitemchunk', - name='uid', - field=models.CharField(db_index=True, max_length=60, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + model_name="collectionitemchunk", + name="uid", + field=models.CharField( + db_index=True, + max_length=60, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]{20,}$") + ], + ), ), migrations.AlterField( - model_name='collectionitemrevision', - name='uid', - field=models.CharField(db_index=True, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + model_name="collectionitemrevision", + name="uid", + field=models.CharField( + db_index=True, + max_length=43, + unique=True, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]{20,}$") + ], + ), ), migrations.AlterField( - model_name='stoken', - name='uid', - field=models.CharField(db_index=True, default=django_etebase.models.generate_stoken_uid, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + model_name="stoken", + name="uid", + field=models.CharField( + db_index=True, + default=django_etebase.models.generate_stoken_uid, + max_length=43, + unique=True, + validators=[ + django.core.validators.RegexValidator(message="Not a valid UID", regex="^[a-zA-Z0-9\\-_]{20,}$") + ], + ), ), ] diff --git a/django_etebase/migrations/0022_auto_20200804_1059.py b/django_etebase/migrations/0022_auto_20200804_1059.py index c47e562..60af33f 100644 --- a/django_etebase/migrations/0022_auto_20200804_1059.py +++ b/django_etebase/migrations/0022_auto_20200804_1059.py @@ -6,12 +6,9 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0021_auto_20200626_0913'), + ("django_etebase", "0021_auto_20200626_0913"), ] operations = [ - migrations.AlterUniqueTogether( - name='collectionitemchunk', - unique_together={('item', 'uid')}, - ), + migrations.AlterUniqueTogether(name="collectionitemchunk", unique_together={("item", "uid")},), ] diff --git a/django_etebase/migrations/0023_collectionitemchunk_collection.py b/django_etebase/migrations/0023_collectionitemchunk_collection.py index b5d6841..314302f 100644 --- a/django_etebase/migrations/0023_collectionitemchunk_collection.py +++ b/django_etebase/migrations/0023_collectionitemchunk_collection.py @@ -7,13 +7,18 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0022_auto_20200804_1059'), + ("django_etebase", "0022_auto_20200804_1059"), ] operations = [ migrations.AddField( - model_name='collectionitemchunk', - name='collection', - field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etebase.Collection'), + model_name="collectionitemchunk", + name="collection", + field=models.ForeignKey( + null=True, + on_delete=django.db.models.deletion.CASCADE, + related_name="chunks", + to="django_etebase.Collection", + ), ), ] diff --git a/django_etebase/migrations/0024_auto_20200804_1209.py b/django_etebase/migrations/0024_auto_20200804_1209.py index 54c80a3..955a4f9 100644 --- a/django_etebase/migrations/0024_auto_20200804_1209.py +++ b/django_etebase/migrations/0024_auto_20200804_1209.py @@ -4,7 +4,7 @@ from django.db import migrations def change_chunk_to_collections(apps, schema_editor): - CollectionItemChunk = apps.get_model('django_etebase', 'CollectionItemChunk') + CollectionItemChunk = apps.get_model("django_etebase", "CollectionItemChunk") for chunk in CollectionItemChunk.objects.all(): chunk.collection = chunk.item.collection @@ -14,7 +14,7 @@ def change_chunk_to_collections(apps, schema_editor): class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0023_collectionitemchunk_collection'), + ("django_etebase", "0023_collectionitemchunk_collection"), ] operations = [ diff --git a/django_etebase/migrations/0025_auto_20200804_1216.py b/django_etebase/migrations/0025_auto_20200804_1216.py index 8849f53..91bf4c8 100644 --- a/django_etebase/migrations/0025_auto_20200804_1216.py +++ b/django_etebase/migrations/0025_auto_20200804_1216.py @@ -7,21 +7,17 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0024_auto_20200804_1209'), + ("django_etebase", "0024_auto_20200804_1209"), ] operations = [ migrations.AlterField( - model_name='collectionitemchunk', - name='collection', - field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='chunks', to='django_etebase.Collection'), - ), - migrations.AlterUniqueTogether( - name='collectionitemchunk', - unique_together={('collection', 'uid')}, - ), - migrations.RemoveField( - model_name='collectionitemchunk', - name='item', + model_name="collectionitemchunk", + name="collection", + field=models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, related_name="chunks", to="django_etebase.Collection" + ), ), + migrations.AlterUniqueTogether(name="collectionitemchunk", unique_together={("collection", "uid")},), + migrations.RemoveField(model_name="collectionitemchunk", name="item",), ] diff --git a/django_etebase/migrations/0026_auto_20200907_0752.py b/django_etebase/migrations/0026_auto_20200907_0752.py index 38c0b92..3283654 100644 --- a/django_etebase/migrations/0026_auto_20200907_0752.py +++ b/django_etebase/migrations/0026_auto_20200907_0752.py @@ -6,18 +6,10 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0025_auto_20200804_1216'), + ("django_etebase", "0025_auto_20200804_1216"), ] operations = [ - migrations.RenameField( - model_name='collectioninvitation', - old_name='accessLevel', - new_name='accessLevelOld', - ), - migrations.RenameField( - model_name='collectionmember', - old_name='accessLevel', - new_name='accessLevelOld', - ), + migrations.RenameField(model_name="collectioninvitation", old_name="accessLevel", new_name="accessLevelOld",), + migrations.RenameField(model_name="collectionmember", old_name="accessLevel", new_name="accessLevelOld",), ] diff --git a/django_etebase/migrations/0027_auto_20200907_0752.py b/django_etebase/migrations/0027_auto_20200907_0752.py index d822d3d..21f607f 100644 --- a/django_etebase/migrations/0027_auto_20200907_0752.py +++ b/django_etebase/migrations/0027_auto_20200907_0752.py @@ -6,18 +6,18 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0026_auto_20200907_0752'), + ("django_etebase", "0026_auto_20200907_0752"), ] operations = [ migrations.AddField( - model_name='collectioninvitation', - name='accessLevel', - field=models.IntegerField(choices=[(0, 'Read Only'), (1, 'Admin'), (2, 'Read Write')], default=0), + model_name="collectioninvitation", + name="accessLevel", + field=models.IntegerField(choices=[(0, "Read Only"), (1, "Admin"), (2, "Read Write")], default=0), ), migrations.AddField( - model_name='collectionmember', - name='accessLevel', - field=models.IntegerField(choices=[(0, 'Read Only'), (1, 'Admin'), (2, 'Read Write')], default=0), + model_name="collectionmember", + name="accessLevel", + field=models.IntegerField(choices=[(0, "Read Only"), (1, "Admin"), (2, "Read Write")], default=0), ), ] diff --git a/django_etebase/migrations/0028_auto_20200907_0754.py b/django_etebase/migrations/0028_auto_20200907_0754.py index cb62e63..24c6246 100644 --- a/django_etebase/migrations/0028_auto_20200907_0754.py +++ b/django_etebase/migrations/0028_auto_20200907_0754.py @@ -6,24 +6,24 @@ from django_etebase.models import AccessLevels def change_access_level_to_int(apps, schema_editor): - CollectionMember = apps.get_model('django_etebase', 'CollectionMember') - CollectionInvitation = apps.get_model('django_etebase', 'CollectionInvitation') + CollectionMember = apps.get_model("django_etebase", "CollectionMember") + CollectionInvitation = apps.get_model("django_etebase", "CollectionInvitation") for member in CollectionMember.objects.all(): - if member.accessLevelOld == 'adm': + if member.accessLevelOld == "adm": member.accessLevel = AccessLevels.ADMIN - elif member.accessLevelOld == 'rw': + elif member.accessLevelOld == "rw": member.accessLevel = AccessLevels.READ_WRITE - elif member.accessLevelOld == 'ro': + elif member.accessLevelOld == "ro": member.accessLevel = AccessLevels.READ_ONLY member.save() for invitation in CollectionInvitation.objects.all(): - if invitation.accessLevelOld == 'adm': + if invitation.accessLevelOld == "adm": invitation.accessLevel = AccessLevels.ADMIN - elif invitation.accessLevelOld == 'rw': + elif invitation.accessLevelOld == "rw": invitation.accessLevel = AccessLevels.READ_WRITE - elif invitation.accessLevelOld == 'ro': + elif invitation.accessLevelOld == "ro": invitation.accessLevel = AccessLevels.READ_ONLY invitation.save() @@ -31,7 +31,7 @@ def change_access_level_to_int(apps, schema_editor): class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0027_auto_20200907_0752'), + ("django_etebase", "0027_auto_20200907_0752"), ] operations = [ diff --git a/django_etebase/migrations/0029_auto_20200907_0801.py b/django_etebase/migrations/0029_auto_20200907_0801.py index 7cd54d4..f3bfe61 100644 --- a/django_etebase/migrations/0029_auto_20200907_0801.py +++ b/django_etebase/migrations/0029_auto_20200907_0801.py @@ -6,16 +6,10 @@ from django.db import migrations class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0028_auto_20200907_0754'), + ("django_etebase", "0028_auto_20200907_0754"), ] operations = [ - migrations.RemoveField( - model_name='collectioninvitation', - name='accessLevelOld', - ), - migrations.RemoveField( - model_name='collectionmember', - name='accessLevelOld', - ), + migrations.RemoveField(model_name="collectioninvitation", name="accessLevelOld",), + migrations.RemoveField(model_name="collectionmember", name="accessLevelOld",), ] diff --git a/django_etebase/migrations/0030_auto_20200922_0832.py b/django_etebase/migrations/0030_auto_20200922_0832.py index d5fa95d..a689251 100644 --- a/django_etebase/migrations/0030_auto_20200922_0832.py +++ b/django_etebase/migrations/0030_auto_20200922_0832.py @@ -7,13 +7,18 @@ import django.db.models.deletion class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0029_auto_20200907_0801'), + ("django_etebase", "0029_auto_20200907_0801"), ] operations = [ migrations.AlterField( - model_name='collection', - name='main_item', - field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='parent', to='django_etebase.collectionitem'), + model_name="collection", + name="main_item", + field=models.OneToOneField( + null=True, + on_delete=django.db.models.deletion.SET_NULL, + related_name="parent", + to="django_etebase.collectionitem", + ), ), ] diff --git a/django_etebase/migrations/0031_auto_20201013_1336.py b/django_etebase/migrations/0031_auto_20201013_1336.py index ca45dd4..ae6e5e5 100644 --- a/django_etebase/migrations/0031_auto_20201013_1336.py +++ b/django_etebase/migrations/0031_auto_20201013_1336.py @@ -9,21 +9,23 @@ class Migration(migrations.Migration): dependencies = [ migrations.swappable_dependency(settings.AUTH_USER_MODEL), - ('django_etebase', '0030_auto_20200922_0832'), + ("django_etebase", "0030_auto_20200922_0832"), ] operations = [ migrations.CreateModel( - name='CollectionType', + name="CollectionType", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('uid', models.BinaryField(db_index=True, editable=True)), - ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ("uid", models.BinaryField(db_index=True, editable=True)), + ("owner", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], ), migrations.AddField( - model_name='collectionmember', - name='collectionType', - field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.PROTECT, to='django_etebase.collectiontype'), + model_name="collectionmember", + name="collectionType", + field=models.ForeignKey( + null=True, on_delete=django.db.models.deletion.PROTECT, to="django_etebase.collectiontype" + ), ), ] diff --git a/django_etebase/migrations/0032_auto_20201013_1409.py b/django_etebase/migrations/0032_auto_20201013_1409.py index 5594006..2bb3cb0 100644 --- a/django_etebase/migrations/0032_auto_20201013_1409.py +++ b/django_etebase/migrations/0032_auto_20201013_1409.py @@ -6,13 +6,13 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0031_auto_20201013_1336'), + ("django_etebase", "0031_auto_20201013_1336"), ] operations = [ migrations.AlterField( - model_name='collectiontype', - name='uid', + model_name="collectiontype", + name="uid", field=models.BinaryField(db_index=True, editable=True, unique=True), ), ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 0036884..691947d 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -27,7 +27,7 @@ from . import app_settings from .exceptions import EtebaseValidationError -UidValidator = RegexValidator(regex=r'^[a-zA-Z0-9\-_]{20,}$', message='Not a valid UID') +UidValidator = RegexValidator(regex=r"^[a-zA-Z0-9\-_]{20,}$", message="Not a valid UID") class CollectionType(models.Model): @@ -36,7 +36,7 @@ class CollectionType(models.Model): class Collection(models.Model): - main_item = models.OneToOneField('CollectionItem', related_name='parent', null=True, on_delete=models.SET_NULL) + main_item = models.OneToOneField("CollectionItem", related_name="parent", null=True, on_delete=models.SET_NULL) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) def __str__(self): @@ -56,38 +56,45 @@ class Collection(models.Model): @cached_property def stoken(self): - stoken = Stoken.objects.filter( - Q(collectionitemrevision__item__collection=self) | Q(collectionmember__collection=self) - ).order_by('id').last() + stoken = ( + Stoken.objects.filter( + Q(collectionitemrevision__item__collection=self) | Q(collectionmember__collection=self) + ) + .order_by("id") + .last() + ) if stoken is None: - raise Exception('stoken is None. Should never happen') + raise Exception("stoken is None. Should never happen") return stoken.uid def validate_unique(self, exclude=None): super().validate_unique(exclude=exclude) - if exclude is None or 'main_item' in exclude: + if exclude is None or "main_item" in exclude: return - if self.__class__.objects.filter(owner=self.owner, main_item__uid=self.main_item.uid) \ - .exclude(id=self.id).exists(): - raise EtebaseValidationError('unique_uid', 'Collection with this uid already exists', - status_code=status.HTTP_409_CONFLICT) + if ( + self.__class__.objects.filter(owner=self.owner, main_item__uid=self.main_item.uid) + .exclude(id=self.id) + .exists() + ): + raise EtebaseValidationError( + "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT + ) class CollectionItem(models.Model): - uid = models.CharField(db_index=True, blank=False, - max_length=43, validators=[UidValidator]) - collection = models.ForeignKey(Collection, related_name='items', on_delete=models.CASCADE) + uid = models.CharField(db_index=True, blank=False, max_length=43, validators=[UidValidator]) + collection = models.ForeignKey(Collection, related_name="items", on_delete=models.CASCADE) version = models.PositiveSmallIntegerField() encryptionKey = models.BinaryField(editable=True, blank=False, null=True) class Meta: - unique_together = ('uid', 'collection') + unique_together = ("uid", "collection") def __str__(self): - return '{} {}'.format(self.uid, self.collection.uid) + return "{} {}".format(self.uid, self.collection.uid) @cached_property def content(self): @@ -107,53 +114,60 @@ def chunk_directory_path(instance, filename): user_id = col.owner.id uid_prefix = instance.uid[:2] uid_rest = instance.uid[2:] - return Path('user_{}'.format(user_id), col.uid, uid_prefix, uid_rest) + return Path("user_{}".format(user_id), col.uid, uid_prefix, uid_rest) class CollectionItemChunk(models.Model): - uid = models.CharField(db_index=True, blank=False, null=False, - max_length=60, validators=[UidValidator]) - collection = models.ForeignKey(Collection, related_name='chunks', on_delete=models.CASCADE) + uid = models.CharField(db_index=True, blank=False, null=False, max_length=60, validators=[UidValidator]) + collection = models.ForeignKey(Collection, related_name="chunks", on_delete=models.CASCADE) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) def __str__(self): return self.uid class Meta: - unique_together = ('collection', 'uid') + unique_together = ("collection", "uid") def generate_stoken_uid(): - return get_random_string(32, allowed_chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_') + return get_random_string(32, allowed_chars="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_") class Stoken(models.Model): - uid = models.CharField(db_index=True, unique=True, blank=False, null=False, default=generate_stoken_uid, - max_length=43, validators=[UidValidator]) + uid = models.CharField( + db_index=True, + unique=True, + blank=False, + null=False, + default=generate_stoken_uid, + max_length=43, + validators=[UidValidator], + ) class CollectionItemRevision(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) - uid = models.CharField(db_index=True, unique=True, blank=False, null=False, - max_length=43, validators=[UidValidator]) - item = models.ForeignKey(CollectionItem, related_name='revisions', on_delete=models.CASCADE) + uid = models.CharField( + db_index=True, unique=True, blank=False, null=False, max_length=43, validators=[UidValidator] + ) + item = models.ForeignKey(CollectionItem, related_name="revisions", on_delete=models.CASCADE) meta = models.BinaryField(editable=True, blank=False, null=False) current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) class Meta: - unique_together = ('item', 'current') + unique_together = ("item", "current") def __str__(self): - return '{} {} current={}'.format(self.uid, self.item.uid, self.current) + return "{} {} current={}".format(self.uid, self.item.uid, self.current) class RevisionChunkRelation(models.Model): - chunk = models.ForeignKey(CollectionItemChunk, related_name='revisions_relation', on_delete=models.CASCADE) - revision = models.ForeignKey(CollectionItemRevision, related_name='chunks_relation', on_delete=models.CASCADE) + chunk = models.ForeignKey(CollectionItemChunk, related_name="revisions_relation", on_delete=models.CASCADE) + revision = models.ForeignKey(CollectionItemRevision, related_name="chunks_relation", on_delete=models.CASCADE) class Meta: - ordering = ('id', ) + ordering = ("id",) class AccessLevels(models.IntegerChoices): @@ -164,28 +178,22 @@ class AccessLevels(models.IntegerChoices): class CollectionMember(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT, null=True) - collection = models.ForeignKey(Collection, related_name='members', on_delete=models.CASCADE) + collection = models.ForeignKey(Collection, related_name="members", on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) collectionType = models.ForeignKey(CollectionType, on_delete=models.PROTECT, null=True) - accessLevel = models.IntegerField( - choices=AccessLevels.choices, - default=AccessLevels.READ_ONLY, - ) + accessLevel = models.IntegerField(choices=AccessLevels.choices, default=AccessLevels.READ_ONLY,) class Meta: - unique_together = ('user', 'collection') + unique_together = ("user", "collection") def __str__(self): - return '{} {}'.format(self.collection.uid, self.user) + return "{} {}".format(self.collection.uid, self.user) def revoke(self): with transaction.atomic(): CollectionMemberRemoved.objects.update_or_create( - collection=self.collection, user=self.user, - defaults={ - 'stoken': Stoken.objects.create(), - }, + collection=self.collection, user=self.user, defaults={"stoken": Stoken.objects.create(),}, ) self.delete() @@ -193,36 +201,32 @@ class CollectionMember(models.Model): class CollectionMemberRemoved(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT, null=True) - collection = models.ForeignKey(Collection, related_name='removed_members', on_delete=models.CASCADE) + collection = models.ForeignKey(Collection, related_name="removed_members", on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) class Meta: - unique_together = ('user', 'collection') + unique_together = ("user", "collection") def __str__(self): - return '{} {}'.format(self.collection.uid, self.user) + return "{} {}".format(self.collection.uid, self.user) class CollectionInvitation(models.Model): - uid = models.CharField(db_index=True, blank=False, null=False, - max_length=43, validators=[UidValidator]) + uid = models.CharField(db_index=True, blank=False, null=False, max_length=43, validators=[UidValidator]) version = models.PositiveSmallIntegerField(default=1) fromMember = models.ForeignKey(CollectionMember, on_delete=models.CASCADE) # FIXME: make sure to delete all invitations for the same collection once one is accepted # Make sure to not allow invitations if already a member - user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name='incoming_invitations', on_delete=models.CASCADE) + user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name="incoming_invitations", on_delete=models.CASCADE) signedEncryptionKey = models.BinaryField(editable=False, blank=False, null=False) - accessLevel = models.IntegerField( - choices=AccessLevels.choices, - default=AccessLevels.READ_ONLY, - ) + accessLevel = models.IntegerField(choices=AccessLevels.choices, default=AccessLevels.READ_ONLY,) class Meta: - unique_together = ('user', 'fromMember') + unique_together = ("user", "fromMember") def __str__(self): - return '{} {}'.format(self.fromMember.collection.uid, self.user) + return "{} {}".format(self.fromMember.collection.uid, self.user) @cached_property def collection(self): diff --git a/django_etebase/parsers.py b/django_etebase/parsers.py index 1ca1a70..c7fe58c 100644 --- a/django_etebase/parsers.py +++ b/django_etebase/parsers.py @@ -5,11 +5,12 @@ class ChunkUploadParser(FileUploadParser): """ Parser for chunk upload data. """ - media_type = 'application/octet-stream' + + media_type = "application/octet-stream" def get_filename(self, stream, media_type, parser_context): """ Detects the uploaded file name. """ - view = parser_context['view'] - return parser_context['kwargs'][view.lookup_field] + view = parser_context["view"] + return parser_context["kwargs"][view.lookup_field] diff --git a/django_etebase/permissions.py b/django_etebase/permissions.py index c624404..3c77d06 100644 --- a/django_etebase/permissions.py +++ b/django_etebase/permissions.py @@ -25,13 +25,14 @@ class IsCollectionAdmin(permissions.BasePermission): """ Custom permission to only allow owners of a collection to view it """ + message = { - 'detail': 'Only collection admins can perform this operation.', - 'code': 'admin_access_required', + "detail": "Only collection admins can perform this operation.", + "code": "admin_access_required", } def has_permission(self, request, view): - collection_uid = view.kwargs['collection_uid'] + collection_uid = view.kwargs["collection_uid"] try: collection = view.get_collection_queryset().get(main_item__uid=collection_uid) return is_collection_admin(collection, request.user) @@ -44,13 +45,14 @@ class IsCollectionAdminOrReadOnly(permissions.BasePermission): """ Custom permission to only allow owners of a collection to edit it """ + message = { - 'detail': 'Only collection admins can edit collections.', - 'code': 'admin_access_required', + "detail": "Only collection admins can edit collections.", + "code": "admin_access_required", } def has_permission(self, request, view): - collection_uid = view.kwargs.get('collection_uid', None) + collection_uid = view.kwargs.get("collection_uid", None) # Allow creating new collections if collection_uid is None: @@ -71,13 +73,14 @@ class HasWriteAccessOrReadOnly(permissions.BasePermission): """ Custom permission to restrict write """ + message = { - 'detail': 'You need write access to write to this collection', - 'code': 'no_write_access', + "detail": "You need write access to write to this collection", + "code": "no_write_access", } def has_permission(self, request, view): - collection_uid = view.kwargs['collection_uid'] + collection_uid = view.kwargs["collection_uid"] try: collection = view.get_collection_queryset().get(main_item__uid=collection_uid) if request.method in permissions.SAFE_METHODS: diff --git a/django_etebase/renderers.py b/django_etebase/renderers.py index 43c1a0d..0d359d3 100644 --- a/django_etebase/renderers.py +++ b/django_etebase/renderers.py @@ -15,4 +15,5 @@ class JSONRenderer(DRFJSONRenderer): """ Renderer which serializes to JSON with support for our base64 """ + encoder_class = JSONEncoder diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 4f2d802..ef3b296 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -29,7 +29,7 @@ User = get_user_model() def process_revisions_for_item(item, revision_data): chunks_objs = [] - chunks = revision_data.pop('chunks_relation') + chunks = revision_data.pop("chunks_relation") revision = models.CollectionItemRevision(**revision_data, item=item) revision.validate_unique() # Verify there aren't any validation issues @@ -42,11 +42,11 @@ def process_revisions_for_item(item, revision_data): # If the chunk already exists we assume it's fine. Otherwise, we upload it. if chunk_obj is None: chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) - chunk_obj.chunkFile.save('IGNORED', ContentFile(content)) + chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) chunk_obj.save() else: if chunk_obj is None: - raise EtebaseValidationError('chunk_no_content', 'Tried to create a new chunk without content') + raise EtebaseValidationError("chunk_no_content", "Tried to create a new chunk without content") chunks_objs.append(chunk_obj) @@ -60,7 +60,7 @@ def process_revisions_for_item(item, revision_data): def b64encode(value): - return base64.urlsafe_b64encode(value).decode('ascii').strip('=') + return base64.urlsafe_b64encode(value).decode("ascii").strip("=") def b64decode(data): @@ -85,7 +85,7 @@ class BinaryBase64Field(serializers.Field): class CollectionEncryptionKeyField(BinaryBase64Field): def get_attribute(self, instance): - request = self.context.get('request', None) + request = self.context.get("request", None) if request is not None: return instance.members.get(user=request.user).encryptionKey return None @@ -93,7 +93,7 @@ class CollectionEncryptionKeyField(BinaryBase64Field): class CollectionTypeField(BinaryBase64Field): def get_attribute(self, instance): - request = self.context.get('request', None) + request = self.context.get("request", None) if request is not None: collection_type = instance.members.get(user=request.user).collectionType return collection_type and collection_type.uid @@ -102,7 +102,7 @@ class CollectionTypeField(BinaryBase64Field): class UserSlugRelatedField(serializers.SlugRelatedField): def get_queryset(self): - view = self.context.get('view', None) + view = self.context.get("view", None) return get_user_queryset(super().get_queryset(), view) def __init__(self, **kwargs): @@ -115,15 +115,15 @@ class UserSlugRelatedField(serializers.SlugRelatedField): class ChunksField(serializers.RelatedField): def to_representation(self, obj): obj = obj.chunk - if self.context.get('prefetch') == 'auto': - with open(obj.chunkFile.path, 'rb') as f: + if self.context.get("prefetch") == "auto": + with open(obj.chunkFile.path, "rb") as f: return (obj.uid, f.read()) else: - return (obj.uid, ) + return (obj.uid,) def to_internal_value(self, data): if data[0] is None or data[1] is None: - raise EtebaseValidationError('no_null', 'null is not allowed') + raise EtebaseValidationError("no_null", "null is not allowed") return (data[0], b64decode_or_bytes(data[1])) @@ -133,18 +133,12 @@ class BetterErrorsMixin: nice = [] errors = super().errors for error_type in errors: - if error_type == 'non_field_errors': - nice.extend( - self.flatten_errors(None, errors[error_type]) - ) + if error_type == "non_field_errors": + nice.extend(self.flatten_errors(None, errors[error_type])) else: - nice.extend( - self.flatten_errors(error_type, errors[error_type]) - ) + nice.extend(self.flatten_errors(error_type, errors[error_type])) if nice: - return {'code': 'field_errors', - 'detail': 'Field validations failed.', - 'errors': nice} + return {"code": "field_errors", "detail": "Field validations failed.", "errors": nice} return {} def flatten_errors(self, field_name, errors): @@ -155,54 +149,50 @@ class BetterErrorsMixin: ret.extend(self.flatten_errors("{}.{}".format(field_name, error_key), error)) else: for error in errors: - if hasattr(error, 'detail'): + if hasattr(error, "detail"): message = error.detail[0] - elif hasattr(error, 'message'): + elif hasattr(error, "message"): message = error.message else: message = str(error) - ret.append({ - 'field': field_name, - 'code': error.code, - 'detail': message, - }) + ret.append( + {"field": field_name, "code": error.code, "detail": message,} + ) return ret def transform_validation_error(self, prefix, err): - if hasattr(err, 'error_dict'): + if hasattr(err, "error_dict"): errors = self.flatten_errors(prefix, err.error_dict) - elif not hasattr(err, 'message'): + elif not hasattr(err, "message"): errors = self.flatten_errors(prefix, err.error_list) else: raise EtebaseValidationError(err.code, err.message) - raise serializers.ValidationError({ - 'code': 'field_errors', - 'detail': 'Field validations failed.', - 'errors': errors, - }) + raise serializers.ValidationError( + {"code": "field_errors", "detail": "Field validations failed.", "errors": errors,} + ) class CollectionItemChunkSerializer(BetterErrorsMixin, serializers.ModelSerializer): class Meta: model = models.CollectionItemChunk - fields = ('uid', 'chunkFile') + fields = ("uid", "chunkFile") class CollectionItemRevisionSerializer(BetterErrorsMixin, serializers.ModelSerializer): chunks = ChunksField( - source='chunks_relation', + source="chunks_relation", queryset=models.RevisionChunkRelation.objects.all(), - style={'base_template': 'input.html'}, - many=True + style={"base_template": "input.html"}, + many=True, ) meta = BinaryBase64Field() class Meta: model = models.CollectionItemRevision - fields = ('chunks', 'meta', 'uid', 'deleted') + fields = ("chunks", "meta", "uid", "deleted") extra_kwargs = { - 'uid': {'validators': []}, # We deal with it in the serializers + "uid": {"validators": []}, # We deal with it in the serializers } @@ -213,14 +203,14 @@ class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): class Meta: model = models.CollectionItem - fields = ('uid', 'version', 'encryptionKey', 'content', 'etag') + fields = ("uid", "version", "encryptionKey", "content", "etag") def create(self, validated_data): """Function that's called when this serializer creates an item""" - validate_etag = self.context.get('validate_etag', False) - etag = validated_data.pop('etag') - revision_data = validated_data.pop('content') - uid = validated_data.pop('uid') + validate_etag = self.context.get("validate_etag", False) + etag = validated_data.pop("etag") + revision_data = validated_data.pop("content") + uid = validated_data.pop("uid") Model = self.__class__.Meta.model @@ -229,12 +219,15 @@ class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): cur_etag = instance.etag if not created else None # If we are trying to update an up to date item, abort early and consider it a success - if cur_etag == revision_data.get('uid'): + if cur_etag == revision_data.get("uid"): return instance if validate_etag and cur_etag != etag: - raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(cur_etag, etag), - status_code=status.HTTP_409_CONFLICT) + raise EtebaseValidationError( + "wrong_etag", + "Wrong etag. Expected {} got {}".format(cur_etag, etag), + status_code=status.HTTP_409_CONFLICT, + ) if not created: # We don't have to use select_for_update here because the unique constraint on current guards against @@ -260,14 +253,17 @@ class CollectionItemDepSerializer(BetterErrorsMixin, serializers.ModelSerializer class Meta: model = models.CollectionItem - fields = ('uid', 'etag') + fields = ("uid", "etag") def validate(self, data): - item = self.__class__.Meta.model.objects.get(uid=data['uid']) - etag = data['etag'] + item = self.__class__.Meta.model.objects.get(uid=data["uid"]) + etag = data["etag"] if item.etag != etag: - raise EtebaseValidationError('wrong_etag', 'Wrong etag. Expected {} got {}'.format(item.etag, etag), - status_code=status.HTTP_409_CONFLICT) + raise EtebaseValidationError( + "wrong_etag", + "Wrong etag. Expected {} got {}".format(item.etag, etag), + status_code=status.HTTP_409_CONFLICT, + ) return data @@ -277,49 +273,47 @@ class CollectionItemBulkGetSerializer(BetterErrorsMixin, serializers.ModelSerial class Meta: model = models.CollectionItem - fields = ('uid', 'etag') + fields = ("uid", "etag") class CollectionListMultiSerializer(BetterErrorsMixin, serializers.Serializer): - collectionTypes = serializers.ListField( - child=BinaryBase64Field() - ) + collectionTypes = serializers.ListField(child=BinaryBase64Field()) class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): collectionKey = CollectionEncryptionKeyField() collectionType = CollectionTypeField() - accessLevel = serializers.SerializerMethodField('get_access_level_from_context') + accessLevel = serializers.SerializerMethodField("get_access_level_from_context") stoken = serializers.CharField(read_only=True) - item = CollectionItemSerializer(many=False, source='main_item') + item = CollectionItemSerializer(many=False, source="main_item") class Meta: model = models.Collection - fields = ('item', 'accessLevel', 'collectionKey', 'collectionType', 'stoken') + fields = ("item", "accessLevel", "collectionKey", "collectionType", "stoken") def get_access_level_from_context(self, obj): - request = self.context.get('request', None) + request = self.context.get("request", None) if request is not None: return obj.members.get(user=request.user).accessLevel return None def create(self, validated_data): """Function that's called when this serializer creates an item""" - collection_key = validated_data.pop('collectionKey') - collection_type = validated_data.pop('collectionType') + collection_key = validated_data.pop("collectionKey") + collection_type = validated_data.pop("collectionType") - user = validated_data.get('owner') - main_item_data = validated_data.pop('main_item') - etag = main_item_data.pop('etag') - revision_data = main_item_data.pop('content') + user = validated_data.get("owner") + main_item_data = validated_data.pop("main_item") + etag = main_item_data.pop("etag") + revision_data = main_item_data.pop("content") instance = self.__class__.Meta.model(**validated_data) with transaction.atomic(): _ = self.__class__.Meta.model.objects.select_for_update().filter(owner=user) if etag is not None: - raise EtebaseValidationError('bad_etag', 'etag is not null') + raise EtebaseValidationError("bad_etag", "etag is not null") instance.save() main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) @@ -333,13 +327,14 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) - models.CollectionMember(collection=instance, - stoken=models.Stoken.objects.create(), - user=user, - accessLevel=models.AccessLevels.ADMIN, - encryptionKey=collection_key, - collectionType=collection_type_obj, - ).save() + models.CollectionMember( + collection=instance, + stoken=models.Stoken.objects.create(), + user=user, + accessLevel=models.AccessLevels.ADMIN, + encryptionKey=collection_key, + collectionType=collection_type_obj, + ).save() return instance @@ -348,15 +343,11 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField( - source='user', - read_only=True, - style={'base_template': 'input.html'}, - ) + username = UserSlugRelatedField(source="user", read_only=True, style={"base_template": "input.html"},) class Meta: model = models.CollectionMember - fields = ('username', 'accessLevel') + fields = ("username", "accessLevel") def create(self, validated_data): raise NotImplementedError() @@ -364,7 +355,7 @@ class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer) def update(self, instance, validated_data): with transaction.atomic(): # We only allow updating accessLevel - access_level = validated_data.pop('accessLevel') + access_level = validated_data.pop("accessLevel") if instance.accessLevel != access_level: instance.stoken = models.Stoken.objects.create() instance.accessLevel = access_level @@ -374,31 +365,35 @@ class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer) class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField( - source='user', - queryset=User.objects, - style={'base_template': 'input.html'}, - ) - collection = serializers.CharField(source='collection.uid') - fromUsername = BinaryBase64Field(source='fromMember.user.username', read_only=True) - fromPubkey = BinaryBase64Field(source='fromMember.user.userinfo.pubkey', read_only=True) + username = UserSlugRelatedField(source="user", queryset=User.objects, style={"base_template": "input.html"},) + collection = serializers.CharField(source="collection.uid") + fromUsername = BinaryBase64Field(source="fromMember.user.username", read_only=True) + fromPubkey = BinaryBase64Field(source="fromMember.user.userinfo.pubkey", read_only=True) signedEncryptionKey = BinaryBase64Field() class Meta: model = models.CollectionInvitation - fields = ('username', 'uid', 'collection', 'signedEncryptionKey', 'accessLevel', - 'fromUsername', 'fromPubkey', 'version') + fields = ( + "username", + "uid", + "collection", + "signedEncryptionKey", + "accessLevel", + "fromUsername", + "fromPubkey", + "version", + ) def validate_user(self, value): - request = self.context['request'] + request = self.context["request"] if request.user.username == value.lower(): - raise EtebaseValidationError('no_self_invite', 'Inviting yourself is not allowed') + raise EtebaseValidationError("no_self_invite", "Inviting yourself is not allowed") return value def create(self, validated_data): - request = self.context['request'] - collection = validated_data.pop('collection') + request = self.context["request"] + collection = validated_data.pop("collection") member = collection.members.get(user=request.user) @@ -406,12 +401,12 @@ class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSeriali try: return type(self).Meta.model.objects.create(**validated_data, fromMember=member) except IntegrityError: - raise EtebaseValidationError('invitation_exists', 'Invitation already exists') + raise EtebaseValidationError("invitation_exists", "Invitation already exists") def update(self, instance, validated_data): with transaction.atomic(): - instance.accessLevel = validated_data.pop('accessLevel') - instance.signedEncryptionKey = validated_data.pop('signedEncryptionKey') + instance.accessLevel = validated_data.pop("accessLevel") + instance.signedEncryptionKey = validated_data.pop("signedEncryptionKey") instance.save() return instance @@ -424,9 +419,9 @@ class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): def create(self, validated_data): with transaction.atomic(): - invitation = self.context['invitation'] - encryption_key = validated_data.get('encryptionKey') - collection_type = validated_data.pop('collectionType') + invitation = self.context["invitation"] + encryption_key = validated_data.get("encryptionKey") + collection_type = validated_data.pop("collectionType") user = invitation.user collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) @@ -438,10 +433,11 @@ class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): accessLevel=invitation.accessLevel, encryptionKey=encryption_key, collectionType=collection_type_obj, - ) + ) models.CollectionMemberRemoved.objects.filter( - user=invitation.user, collection=invitation.collection).delete() + user=invitation.user, collection=invitation.collection + ).delete() invitation.delete() @@ -452,12 +448,12 @@ class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): class UserSerializer(BetterErrorsMixin, serializers.ModelSerializer): - pubkey = BinaryBase64Field(source='userinfo.pubkey') - encryptedContent = BinaryBase64Field(source='userinfo.encryptedContent') + pubkey = BinaryBase64Field(source="userinfo.pubkey") + encryptedContent = BinaryBase64Field(source="userinfo.encryptedContent") class Meta: model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, 'pubkey', 'encryptedContent') + fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, "pubkey", "encryptedContent") class UserInfoPubkeySerializer(BetterErrorsMixin, serializers.ModelSerializer): @@ -465,7 +461,7 @@ class UserInfoPubkeySerializer(BetterErrorsMixin, serializers.ModelSerializer): class Meta: model = models.UserInfo - fields = ('pubkey', ) + fields = ("pubkey",) class UserSignupSerializer(BetterErrorsMixin, serializers.ModelSerializer): @@ -473,7 +469,7 @@ class UserSignupSerializer(BetterErrorsMixin, serializers.ModelSerializer): model = User fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) extra_kwargs = { - 'username': {'validators': []}, # We specifically validate in SignupSerializer + "username": {"validators": []}, # We specifically validate in SignupSerializer } @@ -481,6 +477,7 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): """Used both for creating new accounts and setting up existing ones for the first time. When setting up existing ones the email is ignored." """ + user = UserSignupSerializer(many=False) salt = BinaryBase64Field() loginPubkey = BinaryBase64Field() @@ -489,27 +486,27 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): def create(self, validated_data): """Function that's called when this serializer creates an item""" - user_data = validated_data.pop('user') + user_data = validated_data.pop("user") with transaction.atomic(): try: - view = self.context.get('view', None) + view = self.context.get("view", None) user_queryset = get_user_queryset(User.objects.all(), view) - instance = user_queryset.get(**{User.USERNAME_FIELD: user_data['username'].lower()}) + instance = user_queryset.get(**{User.USERNAME_FIELD: user_data["username"].lower()}) except User.DoesNotExist: # Create the user and save the casing the user chose as the first name try: - instance = create_user(**user_data, password=None, first_name=user_data['username'], view=view) + instance = create_user(**user_data, password=None, first_name=user_data["username"], view=view) instance.clean_fields() except EtebaseValidationError as e: raise e except django_exceptions.ValidationError as e: self.transform_validation_error("user", e) except Exception as e: - raise EtebaseValidationError('generic', str(e)) + raise EtebaseValidationError("generic", str(e)) - if hasattr(instance, 'userinfo'): - raise EtebaseValidationError('user_exists', 'User already exists', status_code=status.HTTP_409_CONFLICT) + if hasattr(instance, "userinfo"): + raise EtebaseValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) models.UserInfo.objects.create(**validated_data, owner=instance) @@ -558,15 +555,15 @@ class AuthenticationChangePasswordInnerSerializer(AuthenticationLoginInnerSerial class Meta: model = models.UserInfo - fields = ('loginPubkey', 'encryptedContent') + fields = ("loginPubkey", "encryptedContent") def create(self, validated_data): raise NotImplementedError() def update(self, instance, validated_data): with transaction.atomic(): - instance.loginPubkey = validated_data.pop('loginPubkey') - instance.encryptedContent = validated_data.pop('encryptedContent') + instance.loginPubkey = validated_data.pop("loginPubkey") + instance.encryptedContent = validated_data.pop("encryptedContent") instance.save() return instance diff --git a/django_etebase/signals.py b/django_etebase/signals.py index 03dbed5..0fc3e80 100644 --- a/django_etebase/signals.py +++ b/django_etebase/signals.py @@ -1,3 +1,3 @@ from django.dispatch import Signal -user_signed_up = Signal(providing_args=['request', 'user']) +user_signed_up = Signal(providing_args=["request", "user"]) diff --git a/django_etebase/token_auth/apps.py b/django_etebase/token_auth/apps.py index 118b872..a0e98be 100644 --- a/django_etebase/token_auth/apps.py +++ b/django_etebase/token_auth/apps.py @@ -2,4 +2,4 @@ from django.apps import AppConfig class TokenAuthConfig(AppConfig): - name = 'django_etebase.token_auth' + name = "django_etebase.token_auth" diff --git a/django_etebase/token_auth/authentication.py b/django_etebase/token_auth/authentication.py index 432c8cf..7e84956 100644 --- a/django_etebase/token_auth/authentication.py +++ b/django_etebase/token_auth/authentication.py @@ -12,19 +12,19 @@ MIN_REFRESH_INTERVAL = 60 class TokenAuthentication(DRFTokenAuthentication): - keyword = 'Token' + keyword = "Token" model = AuthToken def authenticate_credentials(self, key): - msg = _('Invalid token.') + msg = _("Invalid token.") model = self.get_model() try: - token = model.objects.select_related('user').get(key=key) + token = model.objects.select_related("user").get(key=key) except model.DoesNotExist: raise exceptions.AuthenticationFailed(msg) if not token.user.is_active: - raise exceptions.AuthenticationFailed(_('User inactive or deleted.')) + raise exceptions.AuthenticationFailed(_("User inactive or deleted.")) if token.expiry is not None: if token.expiry < timezone.now(): @@ -43,4 +43,4 @@ class TokenAuthentication(DRFTokenAuthentication): delta = (new_expiry - current_expiry).total_seconds() if delta > MIN_REFRESH_INTERVAL: auth_token.expiry = new_expiry - auth_token.save(update_fields=('expiry',)) + auth_token.save(update_fields=("expiry",)) diff --git a/django_etebase/token_auth/migrations/0001_initial.py b/django_etebase/token_auth/migrations/0001_initial.py index 5a47366..660b38c 100644 --- a/django_etebase/token_auth/migrations/0001_initial.py +++ b/django_etebase/token_auth/migrations/0001_initial.py @@ -16,13 +16,23 @@ class Migration(migrations.Migration): operations = [ migrations.CreateModel( - name='AuthToken', + name="AuthToken", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('key', models.CharField(db_index=True, default=token_auth_models.generate_key, max_length=40, unique=True)), - ('created', models.DateTimeField(auto_now_add=True)), - ('expiry', models.DateTimeField(blank=True, default=token_auth_models.get_default_expiry, null=True)), - ('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='auth_token_set', to=settings.AUTH_USER_MODEL)), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ( + "key", + models.CharField(db_index=True, default=token_auth_models.generate_key, max_length=40, unique=True), + ), + ("created", models.DateTimeField(auto_now_add=True)), + ("expiry", models.DateTimeField(blank=True, default=token_auth_models.get_default_expiry, null=True)), + ( + "user", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="auth_token_set", + to=settings.AUTH_USER_MODEL, + ), + ), ], ), ] diff --git a/django_etebase/token_auth/models.py b/django_etebase/token_auth/models.py index 0fe4766..ac1efff 100644 --- a/django_etebase/token_auth/models.py +++ b/django_etebase/token_auth/models.py @@ -17,10 +17,9 @@ def get_default_expiry(): class AuthToken(models.Model): key = models.CharField(max_length=40, unique=True, db_index=True, default=generate_key) - user = models.ForeignKey(User, null=False, blank=False, - related_name='auth_token_set', on_delete=models.CASCADE) + user = models.ForeignKey(User, null=False, blank=False, related_name="auth_token_set", on_delete=models.CASCADE) created = models.DateTimeField(auto_now_add=True) expiry = models.DateTimeField(null=True, blank=True, default=get_default_expiry) def __str__(self): - return '{}: {}'.format(self.key, self.user) + return "{}: {}".format(self.key, self.user) diff --git a/django_etebase/urls.py b/django_etebase/urls.py index f6d982e..01797c1 100644 --- a/django_etebase/urls.py +++ b/django_etebase/urls.py @@ -7,24 +7,24 @@ from rest_framework_nested import routers from django_etebase import views router = routers.DefaultRouter() -router.register(r'collection', views.CollectionViewSet) -router.register(r'authentication', views.AuthenticationViewSet, basename='authentication') -router.register(r'invitation/incoming', views.InvitationIncomingViewSet, basename='invitation_incoming') -router.register(r'invitation/outgoing', views.InvitationOutgoingViewSet, basename='invitation_outgoing') +router.register(r"collection", views.CollectionViewSet) +router.register(r"authentication", views.AuthenticationViewSet, basename="authentication") +router.register(r"invitation/incoming", views.InvitationIncomingViewSet, basename="invitation_incoming") +router.register(r"invitation/outgoing", views.InvitationOutgoingViewSet, basename="invitation_outgoing") -collections_router = routers.NestedSimpleRouter(router, r'collection', lookup='collection') -collections_router.register(r'item', views.CollectionItemViewSet, basename='collection_item') -collections_router.register(r'member', views.CollectionMemberViewSet, basename='collection_member') +collections_router = routers.NestedSimpleRouter(router, r"collection", lookup="collection") +collections_router.register(r"item", views.CollectionItemViewSet, basename="collection_item") +collections_router.register(r"member", views.CollectionMemberViewSet, basename="collection_member") -item_router = routers.NestedSimpleRouter(collections_router, r'item', lookup='collection_item') -item_router.register(r'chunk', views.CollectionItemChunkViewSet, basename='collection_items_chunk') +item_router = routers.NestedSimpleRouter(collections_router, r"item", lookup="collection_item") +item_router.register(r"chunk", views.CollectionItemChunkViewSet, basename="collection_items_chunk") if settings.DEBUG: - router.register(r'test/authentication', views.TestAuthenticationViewSet, basename='test_authentication') + router.register(r"test/authentication", views.TestAuthenticationViewSet, basename="test_authentication") -app_name = 'django_etebase' +app_name = "django_etebase" urlpatterns = [ - path('v1/', include(router.urls)), - path('v1/', include(collections_router.urls)), - path('v1/', include(item_router.urls)), + path("v1/", include(router.urls)), + path("v1/", include(collections_router.urls)), + path("v1/", include(item_router.urls)), ] diff --git a/django_etebase/utils.py b/django_etebase/utils.py index 1351f9b..e496a77 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -18,9 +18,9 @@ def create_user(*args, **kwargs): custom_func = app_settings.CREATE_USER_FUNC if custom_func is not None: return custom_func(*args, **kwargs) - _ = kwargs.pop('view') + _ = kwargs.pop("view") return User.objects.create_user(*args, **kwargs) def create_user_blocked(*args, **kwargs): - raise PermissionDenied('Signup is disabled for this server. Please refer to the README for more information.') + raise PermissionDenied("Signup is disabled for this server. Please refer to the README for more information.") diff --git a/django_etebase/views.py b/django_etebase/views.py index 2dc7adf..7dd7526 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -45,34 +45,34 @@ from .drf_msgpack.renderers import MessagePackRenderer from . import app_settings, permissions from .renderers import JSONRenderer from .models import ( - Collection, - CollectionItem, - CollectionItemRevision, - CollectionMember, - CollectionMemberRemoved, - CollectionInvitation, - Stoken, - UserInfo, - ) + Collection, + CollectionItem, + CollectionItemRevision, + CollectionMember, + CollectionMemberRemoved, + CollectionInvitation, + Stoken, + UserInfo, +) from .serializers import ( - AuthenticationChangePasswordInnerSerializer, - AuthenticationSignupSerializer, - AuthenticationLoginChallengeSerializer, - AuthenticationLoginSerializer, - AuthenticationLoginInnerSerializer, - CollectionSerializer, - CollectionItemSerializer, - CollectionItemBulkGetSerializer, - CollectionItemDepSerializer, - CollectionItemRevisionSerializer, - CollectionItemChunkSerializer, - CollectionListMultiSerializer, - CollectionMemberSerializer, - CollectionInvitationSerializer, - InvitationAcceptSerializer, - UserInfoPubkeySerializer, - UserSerializer, - ) + AuthenticationChangePasswordInnerSerializer, + AuthenticationSignupSerializer, + AuthenticationLoginChallengeSerializer, + AuthenticationLoginSerializer, + AuthenticationLoginInnerSerializer, + CollectionSerializer, + CollectionItemSerializer, + CollectionItemBulkGetSerializer, + CollectionItemDepSerializer, + CollectionItemRevisionSerializer, + CollectionItemChunkSerializer, + CollectionListMultiSerializer, + CollectionMemberSerializer, + CollectionInvitationSerializer, + InvitationAcceptSerializer, + UserInfoPubkeySerializer, + UserSerializer, +) from .utils import get_user_queryset from .exceptions import EtebaseValidationError from .parsers import ChunkUploadParser @@ -99,8 +99,8 @@ class BaseViewSet(viewsets.ModelViewSet): def get_serializer_class(self): serializer_class = self.serializer_class - if self.request.method == 'PUT': - serializer_class = getattr(self, 'serializer_update_class', serializer_class) + if self.request.method == "PUT": + serializer_class = getattr(self, "serializer_update_class", serializer_class) return serializer_class @@ -109,7 +109,7 @@ class BaseViewSet(viewsets.ModelViewSet): return queryset.filter(members__user=user) def get_stoken_obj_id(self, request): - return request.GET.get('stoken', None) + return request.GET.get("stoken", None) def get_stoken_obj(self, request): stoken = self.get_stoken_obj_id(request) @@ -118,7 +118,7 @@ class BaseViewSet(viewsets.ModelViewSet): try: return Stoken.objects.get(uid=stoken) except Stoken.DoesNotExist: - raise EtebaseValidationError('bad_stoken', 'Invalid stoken.', status_code=status.HTTP_400_BAD_REQUEST) + raise EtebaseValidationError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) return None @@ -127,7 +127,7 @@ class BaseViewSet(viewsets.ModelViewSet): aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] - queryset = queryset.annotate(max_stoken=max_stoken).order_by('max_stoken') + queryset = queryset.annotate(max_stoken=max_stoken).order_by("max_stoken") if stoken_rev is not None: queryset = queryset.filter(max_stoken__gt=stoken_rev.id) @@ -137,18 +137,18 @@ class BaseViewSet(viewsets.ModelViewSet): def get_queryset_stoken(self, queryset): maxid = -1 for row in queryset: - rowmaxid = getattr(row, 'max_stoken') or -1 + rowmaxid = getattr(row, "max_stoken") or -1 maxid = max(maxid, rowmaxid) new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid) return new_stoken or None def filter_by_stoken_and_limit(self, request, queryset): - limit = int(request.GET.get('limit', 50)) + limit = int(request.GET.get("limit", 50)) queryset, stoken_rev = self.filter_by_stoken(request, queryset) - result = list(queryset[:limit + 1]) + result = list(queryset[: limit + 1]) if len(result) < limit + 1: done = True else: @@ -165,21 +165,21 @@ class BaseViewSet(viewsets.ModelViewSet): serializer = self.get_serializer(queryset, many=True) ret = { - 'data': serializer.data, - 'done': True, # we always return all the items, so it's always done + "data": serializer.data, + "done": True, # we always return all the items, so it's always done } return Response(ret) class CollectionViewSet(BaseViewSet): - allowed_methods = ['GET', 'POST'] - permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly, ) + allowed_methods = ["GET", "POST"] + permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly,) queryset = Collection.objects.all() serializer_class = CollectionSerializer - lookup_field = 'main_item__uid' - lookup_url_kwarg = 'uid' - stoken_id_fields = ['items__revisions__stoken__id', 'members__stoken__id'] + lookup_field = "main_item__uid" + lookup_url_kwarg = "uid" + stoken_id_fields = ["items__revisions__stoken__id", "members__stoken__id"] def get_queryset(self, queryset=None): if queryset is None: @@ -188,8 +188,8 @@ class CollectionViewSet(BaseViewSet): def get_serializer_context(self): context = super().get_serializer_context() - prefetch = self.request.query_params.get('prefetch', 'auto') - context.update({'request': self.request, 'prefetch': prefetch}) + prefetch = self.request.query_params.get("prefetch", "auto") + context.update({"request": self.request, "prefetch": prefetch}) return context def destroy(self, request, uid=None, *args, **kwargs): @@ -213,17 +213,18 @@ class CollectionViewSet(BaseViewSet): queryset = self.get_queryset() return self.list_common(request, queryset, *args, **kwargs) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def list_multi(self, request, *args, **kwargs): serializer = CollectionListMultiSerializer(data=request.data) serializer.is_valid(raise_exception=True) - collection_types = serializer.validated_data['collectionTypes'] + collection_types = serializer.validated_data["collectionTypes"] queryset = self.get_queryset() # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( - Q(members__collectionType__uid__in=collection_types) | Q(members__collectionType__isnull=True)) + Q(members__collectionType__uid__in=collection_types) | Q(members__collectionType__isnull=True) + ) return self.list_common(request, queryset, *args, **kwargs) @@ -234,51 +235,50 @@ class CollectionViewSet(BaseViewSet): serializer = self.get_serializer(result, many=True) ret = { - 'data': serializer.data, - 'stoken': new_stoken, - 'done': done, + "data": serializer.data, + "stoken": new_stoken, + "done": done, } stoken_obj = self.get_stoken_obj(request) if stoken_obj is not None: # FIXME: honour limit? (the limit should be combined for data and this because of stoken) remed_qs = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) - if not ret['done']: + if not ret["done"]: # We only filter by the new_stoken if we are not done. This is because if we are done, the new stoken # can point to the most recent collection change rather than most recent removed membership. remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) - remed = remed_qs.values_list('collection__main_item__uid', flat=True) + remed = remed_qs.values_list("collection__main_item__uid", flat=True) if len(remed) > 0: - ret['removedMemberships'] = [{'uid': x} for x in remed] + ret["removedMemberships"] = [{"uid": x} for x in remed] return Response(ret) class CollectionItemViewSet(BaseViewSet): - allowed_methods = ['GET', 'POST', 'PUT'] - permission_classes = BaseViewSet.permission_classes + (permissions.HasWriteAccessOrReadOnly, ) + allowed_methods = ["GET", "POST", "PUT"] + permission_classes = BaseViewSet.permission_classes + (permissions.HasWriteAccessOrReadOnly,) queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer - lookup_field = 'uid' - stoken_id_fields = ['revisions__stoken__id'] + lookup_field = "uid" + stoken_id_fields = ["revisions__stoken__id"] def get_queryset(self): - collection_uid = self.kwargs['collection_uid'] + collection_uid = self.kwargs["collection_uid"] try: collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) except Collection.DoesNotExist: raise Http404("Collection does not exist") # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') - queryset = type(self).queryset.filter(collection__pk=collection.pk, - revisions__current=True) + queryset = type(self).queryset.filter(collection__pk=collection.pk, revisions__current=True) return queryset def get_serializer_context(self): context = super().get_serializer_context() - prefetch = self.request.query_params.get('prefetch', 'auto') - context.update({'request': self.request, 'prefetch': prefetch}) + prefetch = self.request.query_params.get("prefetch", "auto") + context.update({"request": self.request, "prefetch": prefetch}) return context def create(self, request, collection_uid=None, *args, **kwargs): @@ -298,7 +298,7 @@ class CollectionItemViewSet(BaseViewSet): def list(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset() - if not self.request.query_params.get('withCollection', False): + if not self.request.query_params.get("withCollection", False): queryset = queryset.filter(parent__isnull=True) result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) @@ -307,27 +307,27 @@ class CollectionItemViewSet(BaseViewSet): serializer = self.get_serializer(result, many=True) ret = { - 'data': serializer.data, - 'stoken': new_stoken, - 'done': done, + "data": serializer.data, + "stoken": new_stoken, + "done": done, } return Response(ret) - @action_decorator(detail=True, methods=['GET']) + @action_decorator(detail=True, methods=["GET"]) def revision(self, request, collection_uid=None, uid=None, *args, **kwargs): col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) item = get_object_or_404(col.items, uid=uid) - limit = int(request.GET.get('limit', 50)) - iterator = request.GET.get('iterator', None) + limit = int(request.GET.get("limit", 50)) + iterator = request.GET.get("iterator", None) - queryset = item.revisions.order_by('-id') + queryset = item.revisions.order_by("-id") if iterator is not None: iterator = get_object_or_404(queryset, uid=iterator) queryset = queryset.filter(id__lt=iterator.id) - result = list(queryset[:limit + 1]) + result = list(queryset[: limit + 1]) if len(result) < limit + 1: done = True else: @@ -336,17 +336,17 @@ class CollectionItemViewSet(BaseViewSet): serializer = CollectionItemRevisionSerializer(result, context=self.get_serializer_context(), many=True) - iterator = serializer.data[-1]['uid'] if len(result) > 0 else None + iterator = serializer.data[-1]["uid"] if len(result) > 0 else None ret = { - 'data': serializer.data, - 'iterator': iterator, - 'done': done, + "data": serializer.data, + "iterator": iterator, + "done": done, } return Response(ret) # FIXME: rename to something consistent with what the clients have - maybe list_updates? - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def fetch_updates(self, request, collection_uid=None, *args, **kwargs): queryset = self.get_queryset() @@ -356,79 +356,76 @@ class CollectionItemViewSet(BaseViewSet): item_limit = 200 if len(serializer.validated_data) > item_limit: - content = {'code': 'too_many_items', - 'detail': 'Request has too many items. Limit: {}'. format(item_limit)} + content = {"code": "too_many_items", "detail": "Request has too many items. Limit: {}".format(item_limit)} return Response(content, status=status.HTTP_400_BAD_REQUEST) queryset, stoken_rev = self.filter_by_stoken(request, queryset) - uids, etags = zip(*[(item['uid'], item.get('etag')) for item in serializer.validated_data]) + uids, etags = zip(*[(item["uid"], item.get("etag")) for item in serializer.validated_data]) revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) new_stoken_obj = self.get_queryset_stoken(queryset) new_stoken = new_stoken_obj and new_stoken_obj.uid - stoken = stoken_rev and getattr(stoken_rev, 'uid', None) + stoken = stoken_rev and getattr(stoken_rev, "uid", None) new_stoken = new_stoken or stoken serializer = self.get_serializer(queryset, many=True) ret = { - 'data': serializer.data, - 'stoken': new_stoken, - 'done': True, # we always return all the items, so it's always done + "data": serializer.data, + "stoken": new_stoken, + "done": True, # we always return all the items, so it's always done } return Response(ret) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def batch(self, request, collection_uid=None, *args, **kwargs): return self.transaction(request, collection_uid, validate_etag=False) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def transaction(self, request, collection_uid=None, validate_etag=True, *args, **kwargs): - stoken = request.GET.get('stoken', None) + stoken = request.GET.get("stoken", None) with transaction.atomic(): # We need this for locking on the collection object collection_object = get_object_or_404( self.get_collection_queryset(Collection.objects).select_for_update(), # Lock writes on the collection - main_item__uid=collection_uid) + main_item__uid=collection_uid, + ) if stoken is not None and stoken != collection_object.stoken: - content = {'code': 'stale_stoken', 'detail': 'Stoken is too old'} + content = {"code": "stale_stoken", "detail": "Stoken is too old"} return Response(content, status=status.HTTP_409_CONFLICT) - items = request.data.get('items') - deps = request.data.get('deps', None) + items = request.data.get("items") + deps = request.data.get("deps", None) # FIXME: It should just be one serializer context = self.get_serializer_context() - context.update({'validate_etag': validate_etag}) + context.update({"validate_etag": validate_etag}) serializer = self.get_serializer_class()(data=items, context=context, many=True) deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) ser_valid = serializer.is_valid() - deps_ser_valid = (deps is None or deps_serializer.is_valid()) + deps_ser_valid = deps is None or deps_serializer.is_valid() if ser_valid and deps_ser_valid: items = serializer.save(collection=collection_object) - ret = { - } + ret = {} return Response(ret, status=status.HTTP_200_OK) return Response( - { - "items": serializer.errors, - "deps": deps_serializer.errors if deps is not None else [], - }, - status=status.HTTP_409_CONFLICT) + {"items": serializer.errors, "deps": deps_serializer.errors if deps is not None else [],}, + status=status.HTTP_409_CONFLICT, + ) class CollectionItemChunkViewSet(viewsets.ViewSet): - allowed_methods = ['GET', 'PUT'] + allowed_methods = ["GET", "PUT"] authentication_classes = BaseViewSet.authentication_classes permission_classes = BaseViewSet.permission_classes renderer_classes = BaseViewSet.renderer_classes - parser_classes = (ChunkUploadParser, ) + parser_classes = (ChunkUploadParser,) serializer_class = CollectionItemChunkSerializer - lookup_field = 'uid' + lookup_field = "uid" def get_serializer_class(self): return self.serializer_class @@ -452,13 +449,12 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): serializer.save(collection=col) except IntegrityError: return Response( - {"code": "chunk_exists", "detail": "Chunk already exists."}, - status=status.HTTP_409_CONFLICT + {"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT ) return Response({}, status=status.HTTP_201_CREATED) - @action_decorator(detail=True, methods=['GET']) + @action_decorator(detail=True, methods=["GET"]) def download(self, request, collection_uid=None, collection_item_uid=None, uid=None, *args, **kwargs): import os from django.views.static import serve @@ -476,24 +472,24 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): class CollectionMemberViewSet(BaseViewSet): - allowed_methods = ['GET', 'PUT', 'DELETE'] + allowed_methods = ["GET", "PUT", "DELETE"] our_base_permission_classes = BaseViewSet.permission_classes - permission_classes = our_base_permission_classes + (permissions.IsCollectionAdmin, ) + permission_classes = our_base_permission_classes + (permissions.IsCollectionAdmin,) queryset = CollectionMember.objects.all() serializer_class = CollectionMemberSerializer - lookup_field = f'user__{User.USERNAME_FIELD}__iexact' - lookup_url_kwarg = 'username' - stoken_id_fields = ['stoken__id'] + lookup_field = f"user__{User.USERNAME_FIELD}__iexact" + lookup_url_kwarg = "username" + stoken_id_fields = ["stoken__id"] # FIXME: need to make sure that there's always an admin, and maybe also don't let an owner remove adm access # (if we want to transfer, we need to do that specifically) def get_queryset(self, queryset=None): - collection_uid = self.kwargs['collection_uid'] + collection_uid = self.kwargs["collection_uid"] try: collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) except Collection.DoesNotExist: - raise Http404('Collection does not exist') + raise Http404("Collection does not exist") if queryset is None: queryset = type(self).queryset @@ -502,18 +498,18 @@ class CollectionMemberViewSet(BaseViewSet): # We override this method because we expect the stoken to be called iterator def get_stoken_obj_id(self, request): - return request.GET.get('iterator', None) + return request.GET.get("iterator", None) def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset().order_by('id') + queryset = self.get_queryset().order_by("id") result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) new_stoken = new_stoken_obj and new_stoken_obj.uid serializer = self.get_serializer(result, many=True) ret = { - 'data': serializer.data, - 'iterator': new_stoken, # Here we call it an iterator, it's only stoken for collection/items - 'done': done, + "data": serializer.data, + "iterator": new_stoken, # Here we call it an iterator, it's only stoken for collection/items + "done": done, } return Response(ret) @@ -526,9 +522,9 @@ class CollectionMemberViewSet(BaseViewSet): def perform_destroy(self, instance): instance.revoke() - @action_decorator(detail=False, methods=['POST'], permission_classes=our_base_permission_classes) + @action_decorator(detail=False, methods=["POST"], permission_classes=our_base_permission_classes) def leave(self, request, collection_uid=None, *args, **kwargs): - collection_uid = self.kwargs['collection_uid'] + collection_uid = self.kwargs["collection_uid"] col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) member = col.members.get(user=request.user) @@ -540,20 +536,20 @@ class CollectionMemberViewSet(BaseViewSet): class InvitationBaseViewSet(BaseViewSet): queryset = CollectionInvitation.objects.all() serializer_class = CollectionInvitationSerializer - lookup_field = 'uid' - lookup_url_kwarg = 'invitation_uid' + lookup_field = "uid" + lookup_url_kwarg = "invitation_uid" def list(self, request, collection_uid=None, *args, **kwargs): - limit = int(request.GET.get('limit', 50)) - iterator = request.GET.get('iterator', None) + limit = int(request.GET.get("limit", 50)) + iterator = request.GET.get("iterator", None) - queryset = self.get_queryset().order_by('id') + queryset = self.get_queryset().order_by("id") if iterator is not None: iterator = get_object_or_404(queryset, uid=iterator) queryset = queryset.filter(id__gt=iterator.id) - result = list(queryset[:limit + 1]) + result = list(queryset[: limit + 1]) if len(result) < limit + 1: done = True else: @@ -562,19 +558,19 @@ class InvitationBaseViewSet(BaseViewSet): serializer = self.get_serializer(result, many=True) - iterator = serializer.data[-1]['uid'] if len(result) > 0 else None + iterator = serializer.data[-1]["uid"] if len(result) > 0 else None ret = { - 'data': serializer.data, - 'iterator': iterator, - 'done': done, + "data": serializer.data, + "iterator": iterator, + "done": done, } return Response(ret) class InvitationOutgoingViewSet(InvitationBaseViewSet): - allowed_methods = ['GET', 'POST', 'PUT', 'DELETE'] + allowed_methods = ["GET", "POST", "PUT", "DELETE"] def get_queryset(self, queryset=None): if queryset is None: @@ -585,28 +581,29 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): def create(self, request, *args, **kwargs): serializer = self.get_serializer(data=request.data) serializer.is_valid(raise_exception=True) - collection_uid = serializer.validated_data.get('collection', {}).get('uid') + collection_uid = serializer.validated_data.get("collection", {}).get("uid") try: collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) except Collection.DoesNotExist: - raise Http404('Collection does not exist') + raise Http404("Collection does not exist") - if request.user == serializer.validated_data.get('user'): - content = {'code': 'self_invite', 'detail': 'Inviting yourself is invalid'} + if request.user == serializer.validated_data.get("user"): + content = {"code": "self_invite", "detail": "Inviting yourself is invalid"} return Response(content, status=status.HTTP_400_BAD_REQUEST) if not permissions.is_collection_admin(collection, request.user): - raise PermissionDenied({'code': 'admin_access_required', - 'detail': 'User is not an admin of this collection'}) + raise PermissionDenied( + {"code": "admin_access_required", "detail": "User is not an admin of this collection"} + ) serializer.save(collection=collection) return Response({}, status=status.HTTP_201_CREATED) - @action_decorator(detail=False, allowed_methods=['GET'], methods=['GET']) + @action_decorator(detail=False, allowed_methods=["GET"], methods=["GET"]) def fetch_user_profile(self, request, *args, **kwargs): - username = request.GET.get('username') + username = request.GET.get("username") kwargs = {User.USERNAME_FIELD: username.lower()} user = get_object_or_404(get_user_queryset(User.objects.all(), self), **kwargs) user_info = get_object_or_404(UserInfo.objects.all(), owner=user) @@ -615,7 +612,7 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): class InvitationIncomingViewSet(InvitationBaseViewSet): - allowed_methods = ['GET', 'DELETE'] + allowed_methods = ["GET", "DELETE"] def get_queryset(self, queryset=None): if queryset is None: @@ -623,11 +620,11 @@ class InvitationIncomingViewSet(InvitationBaseViewSet): return queryset.filter(user=self.request.user) - @action_decorator(detail=True, allowed_methods=['POST'], methods=['POST']) + @action_decorator(detail=True, allowed_methods=["POST"], methods=["POST"]) def accept(self, request, invitation_uid=None, *args, **kwargs): invitation = get_object_or_404(self.get_queryset(), uid=invitation_uid) context = self.get_serializer_context() - context.update({'invitation': invitation}) + context.update({"invitation": invitation}) serializer = InvitationAcceptSerializer(data=request.data, context=context) serializer.is_valid(raise_exception=True) @@ -636,36 +633,37 @@ class InvitationIncomingViewSet(InvitationBaseViewSet): class AuthenticationViewSet(viewsets.ViewSet): - allowed_methods = ['POST'] + allowed_methods = ["POST"] authentication_classes = BaseViewSet.authentication_classes renderer_classes = BaseViewSet.renderer_classes parser_classes = BaseViewSet.parser_classes def get_encryption_key(self, salt): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) - return nacl.hash.blake2b(b'', key=key, salt=salt[:nacl.hash.BLAKE2B_SALTBYTES], person=b'etebase-auth', - encoder=nacl.encoding.RawEncoder) + return nacl.hash.blake2b( + b"", + key=key, + salt=salt[: nacl.hash.BLAKE2B_SALTBYTES], + person=b"etebase-auth", + encoder=nacl.encoding.RawEncoder, + ) def get_queryset(self): return get_user_queryset(User.objects.all(), self) def get_serializer_context(self): - return { - 'request': self.request, - 'format': self.format_kwarg, - 'view': self - } + return {"request": self.request, "format": self.format_kwarg, "view": self} def login_response_data(self, user): return { - 'token': AuthToken.objects.create(user=user).key, - 'user': UserSerializer(user).data, + "token": AuthToken.objects.create(user=user).key, + "user": UserSerializer(user).data, } def list(self, request, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def signup(self, request, *args, **kwargs): serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) serializer.is_valid(raise_exception=True) @@ -677,23 +675,23 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(data, status=status.HTTP_201_CREATED) def get_login_user(self, username): - kwargs = {User.USERNAME_FIELD + '__iexact': username.lower()} + kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} try: user = self.get_queryset().get(**kwargs) - if not hasattr(user, 'userinfo'): - raise AuthenticationFailed({'code': 'user_not_init', 'detail': 'User not properly init'}) + if not hasattr(user, "userinfo"): + raise AuthenticationFailed({"code": "user_not_init", "detail": "User not properly init"}) return user except User.DoesNotExist: - raise AuthenticationFailed({'code': 'user_not_found', 'detail': 'User not found'}) + raise AuthenticationFailed({"code": "user_not_found", "detail": "User not found"}) def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): from datetime import datetime - username = validated_data.get('username') + username = validated_data.get("username") user = self.get_login_user(username) - host = validated_data['host'] - challenge = validated_data['challenge'] - action = validated_data['action'] + host = validated_data["host"] + challenge = validated_data["challenge"] + action = validated_data["action"] salt = bytes(user.userinfo.salt) enc_key = self.get_encryption_key(salt) @@ -702,17 +700,17 @@ class AuthenticationViewSet(viewsets.ViewSet): challenge_data = msgpack_decode(box.decrypt(challenge)) now = int(datetime.now().timestamp()) if action != expected_action: - content = {'code': 'wrong_action', 'detail': 'Expected "{}" but got something else'.format(expected_action)} + content = {"code": "wrong_action", "detail": 'Expected "{}" but got something else'.format(expected_action)} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif now - challenge_data['timestamp'] > app_settings.CHALLENGE_VALID_SECONDS: - content = {'code': 'challenge_expired', 'detail': 'Login challange has expired'} + elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: + content = {"code": "challenge_expired", "detail": "Login challange has expired"} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif challenge_data['userId'] != user.id: - content = {'code': 'wrong_user', 'detail': 'This challenge is for the wrong user'} + elif challenge_data["userId"] != user.id: + content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host.split(':', 1)[0] != request.get_host(): + elif not settings.DEBUG and host.split(":", 1)[0] != request.get_host(): detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) - content = {'code': 'wrong_host', 'detail': detail} + content = {"code": "wrong_host", "detail": detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) @@ -720,22 +718,24 @@ class AuthenticationViewSet(viewsets.ViewSet): try: verify_key.verify(response_raw, signature) except nacl.exceptions.BadSignatureError: - return Response({'code': 'login_bad_signature', 'detail': 'Wrong password for user.'}, - status=status.HTTP_401_UNAUTHORIZED) + return Response( + {"code": "login_bad_signature", "detail": "Wrong password for user."}, + status=status.HTTP_401_UNAUTHORIZED, + ) return None - @action_decorator(detail=False, methods=['GET']) + @action_decorator(detail=False, methods=["GET"]) def is_etebase(self, request, *args, **kwargs): return Response({}, status=status.HTTP_200_OK) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def login_challenge(self, request, *args, **kwargs): from datetime import datetime serializer = AuthenticationLoginChallengeSerializer(data=request.data) serializer.is_valid(raise_exception=True) - username = serializer.validated_data.get('username') + username = serializer.validated_data.get("username") user = self.get_login_user(username) salt = bytes(user.userinfo.salt) @@ -755,25 +755,26 @@ class AuthenticationViewSet(viewsets.ViewSet): } return Response(ret, status=status.HTTP_200_OK) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def login(self, request, *args, **kwargs): outer_serializer = AuthenticationLoginSerializer(data=request.data) outer_serializer.is_valid(raise_exception=True) - response_raw = outer_serializer.validated_data['response'] + response_raw = outer_serializer.validated_data["response"] response = msgpack_decode(response_raw) - signature = outer_serializer.validated_data['signature'] + signature = outer_serializer.validated_data["signature"] - context = {'host': request.get_host()} + context = {"host": request.get_host()} serializer = AuthenticationLoginInnerSerializer(data=response, context=context) serializer.is_valid(raise_exception=True) bad_login_response = self.validate_login_request( - request, serializer.validated_data, response_raw, signature, "login") + request, serializer.validated_data, response_raw, signature, "login" + ) if bad_login_response is not None: return bad_login_response - username = serializer.validated_data.get('username') + username = serializer.validated_data.get("username") user = self.get_login_user(username) data = self.login_response_data(user) @@ -782,27 +783,28 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response(data, status=status.HTTP_200_OK) - @action_decorator(detail=False, methods=['POST'], permission_classes=[IsAuthenticated]) + @action_decorator(detail=False, methods=["POST"], permission_classes=[IsAuthenticated]) def logout(self, request, *args, **kwargs): request.auth.delete() user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) return Response(status=status.HTTP_204_NO_CONTENT) - @action_decorator(detail=False, methods=['POST'], permission_classes=BaseViewSet.permission_classes) + @action_decorator(detail=False, methods=["POST"], permission_classes=BaseViewSet.permission_classes) def change_password(self, request, *args, **kwargs): outer_serializer = AuthenticationLoginSerializer(data=request.data) outer_serializer.is_valid(raise_exception=True) - response_raw = outer_serializer.validated_data['response'] + response_raw = outer_serializer.validated_data["response"] response = msgpack_decode(response_raw) - signature = outer_serializer.validated_data['signature'] + signature = outer_serializer.validated_data["signature"] - context = {'host': request.get_host()} + context = {"host": request.get_host()} serializer = AuthenticationChangePasswordInnerSerializer(request.user.userinfo, data=response, context=context) serializer.is_valid(raise_exception=True) bad_login_response = self.validate_login_request( - request, serializer.validated_data, response_raw, signature, "changePassword") + request, serializer.validated_data, response_raw, signature, "changePassword" + ) if bad_login_response is not None: return bad_login_response @@ -810,35 +812,32 @@ class AuthenticationViewSet(viewsets.ViewSet): return Response({}, status=status.HTTP_200_OK) - @action_decorator(detail=False, methods=['POST'], permission_classes=[IsAuthenticated]) + @action_decorator(detail=False, methods=["POST"], permission_classes=[IsAuthenticated]) def dashboard_url(self, request, *args, **kwargs): get_dashboard_url = app_settings.DASHBOARD_URL_FUNC if get_dashboard_url is None: - raise EtebaseValidationError('not_supported', 'This server doesn\'t have a user dashboard.', - status_code=status.HTTP_400_BAD_REQUEST) + raise EtebaseValidationError( + "not_supported", "This server doesn't have a user dashboard.", status_code=status.HTTP_400_BAD_REQUEST + ) ret = { - 'url': get_dashboard_url(request, *args, **kwargs), + "url": get_dashboard_url(request, *args, **kwargs), } return Response(ret) class TestAuthenticationViewSet(viewsets.ViewSet): - allowed_methods = ['POST'] + allowed_methods = ["POST"] renderer_classes = BaseViewSet.renderer_classes parser_classes = BaseViewSet.parser_classes def get_serializer_context(self): - return { - 'request': self.request, - 'format': self.format_kwarg, - 'view': self - } + return {"request": self.request, "format": self.format_kwarg, "view": self} def list(self, request, *args, **kwargs): return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - @action_decorator(detail=False, methods=['POST']) + @action_decorator(detail=False, methods=["POST"]) def reset(self, request, *args, **kwargs): # Only run when in DEBUG mode! It's only used for tests if not settings.DEBUG: @@ -846,13 +845,13 @@ class TestAuthenticationViewSet(viewsets.ViewSet): with transaction.atomic(): user_queryset = get_user_queryset(User.objects.all(), self) - user = get_object_or_404(user_queryset, username=request.data.get('user').get('username')) + user = get_object_or_404(user_queryset, username=request.data.get("user").get("username")) # Only allow test users for extra safety - if not getattr(user, User.USERNAME_FIELD).startswith('test_user'): + if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): return HttpResponseBadRequest("Endpoint not allowed for user.") - if hasattr(user, 'userinfo'): + if hasattr(user, "userinfo"): user.userinfo.delete() serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) diff --git a/etebase_server/asgi.py b/etebase_server/asgi.py index 44f1c53..0bf63ec 100644 --- a/etebase_server/asgi.py +++ b/etebase_server/asgi.py @@ -11,6 +11,6 @@ import os from django.core.asgi import get_asgi_application -os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'etebase_server.settings') +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") application = get_asgi_application() diff --git a/etebase_server/settings.py b/etebase_server/settings.py index ee98f55..9baf8d3 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -17,7 +17,7 @@ from .utils import get_secret_from_file # Build paths inside the project like this: os.path.join(BASE_DIR, ...) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -AUTH_USER_MODEL = 'myauth.User' +AUTH_USER_MODEL = "myauth.User" # Quick-start development settings - unsuitable for production @@ -37,10 +37,9 @@ ALLOWED_HOSTS = [] # https://docs.djangoproject.com/en/2.0/ref/settings/#databases DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.sqlite3', - 'NAME': os.environ.get('ETEBASE_DB_PATH', - os.path.join(BASE_DIR, 'db.sqlite3')), + "default": { + "ENGINE": "django.db.backends.sqlite3", + "NAME": os.environ.get("ETEBASE_DB_PATH", os.path.join(BASE_DIR, "db.sqlite3")), } } @@ -48,78 +47,68 @@ DATABASES = { # Application definition INSTALLED_APPS = [ - 'django.contrib.admin', - 'django.contrib.auth', - 'django.contrib.contenttypes', - 'django.contrib.sessions', - 'django.contrib.messages', - 'django.contrib.staticfiles', - 'corsheaders', - 'rest_framework', - 'myauth.apps.MyauthConfig', - 'django_etebase.apps.DjangoEtebaseConfig', - 'django_etebase.token_auth.apps.TokenAuthConfig', + "django.contrib.admin", + "django.contrib.auth", + "django.contrib.contenttypes", + "django.contrib.sessions", + "django.contrib.messages", + "django.contrib.staticfiles", + "corsheaders", + "rest_framework", + "myauth.apps.MyauthConfig", + "django_etebase.apps.DjangoEtebaseConfig", + "django_etebase.token_auth.apps.TokenAuthConfig", ] MIDDLEWARE = [ - 'django.middleware.security.SecurityMiddleware', - 'django.contrib.sessions.middleware.SessionMiddleware', - 'corsheaders.middleware.CorsMiddleware', - 'django.middleware.common.CommonMiddleware', - 'django.middleware.csrf.CsrfViewMiddleware', - 'django.contrib.auth.middleware.AuthenticationMiddleware', - 'django.contrib.messages.middleware.MessageMiddleware', - 'django.middleware.clickjacking.XFrameOptionsMiddleware', + "django.middleware.security.SecurityMiddleware", + "django.contrib.sessions.middleware.SessionMiddleware", + "corsheaders.middleware.CorsMiddleware", + "django.middleware.common.CommonMiddleware", + "django.middleware.csrf.CsrfViewMiddleware", + "django.contrib.auth.middleware.AuthenticationMiddleware", + "django.contrib.messages.middleware.MessageMiddleware", + "django.middleware.clickjacking.XFrameOptionsMiddleware", ] -ROOT_URLCONF = 'etebase_server.urls' +ROOT_URLCONF = "etebase_server.urls" TEMPLATES = [ { - 'BACKEND': 'django.template.backends.django.DjangoTemplates', - 'DIRS': [ - os.path.join(BASE_DIR, 'templates') - ], - 'APP_DIRS': True, - 'OPTIONS': { - 'context_processors': [ - 'django.template.context_processors.debug', - 'django.template.context_processors.request', - 'django.contrib.auth.context_processors.auth', - 'django.contrib.messages.context_processors.messages', + "BACKEND": "django.template.backends.django.DjangoTemplates", + "DIRS": [os.path.join(BASE_DIR, "templates")], + "APP_DIRS": True, + "OPTIONS": { + "context_processors": [ + "django.template.context_processors.debug", + "django.template.context_processors.request", + "django.contrib.auth.context_processors.auth", + "django.contrib.messages.context_processors.messages", ], }, }, ] -WSGI_APPLICATION = 'etebase_server.wsgi.application' +WSGI_APPLICATION = "etebase_server.wsgi.application" # Password validation # https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators AUTH_PASSWORD_VALIDATORS = [ - { - 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', - }, - { - 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', - }, + {"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",}, + {"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",}, + {"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",}, + {"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",}, ] # Internationalization # https://docs.djangoproject.com/en/3.0/topics/i18n/ -LANGUAGE_CODE = 'en-us' +LANGUAGE_CODE = "en-us" -TIME_ZONE = 'UTC' +TIME_ZONE = "UTC" USE_I18N = True @@ -133,18 +122,18 @@ CORS_ORIGIN_ALLOW_ALL = True # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/3.0/howto/static-files/ -STATIC_URL = '/static/' -STATIC_ROOT = os.environ.get('DJANGO_STATIC_ROOT', os.path.join(BASE_DIR, 'static')) +STATIC_URL = "/static/" +STATIC_ROOT = os.environ.get("DJANGO_STATIC_ROOT", os.path.join(BASE_DIR, "static")) -MEDIA_ROOT = os.environ.get('DJANGO_MEDIA_ROOT', os.path.join(BASE_DIR, 'media')) -MEDIA_URL = '/user-media/' +MEDIA_ROOT = os.environ.get("DJANGO_MEDIA_ROOT", os.path.join(BASE_DIR, "media")) +MEDIA_URL = "/user-media/" # Define where to find configuration files config_locations = [ - os.environ.get('ETEBASE_EASY_CONFIG_PATH', ''), - 'etebase-server.ini', - '/etc/etebase-server/etebase-server.ini', + os.environ.get("ETEBASE_EASY_CONFIG_PATH", ""), + "etebase-server.ini", + "/etc/etebase-server/etebase-server.ini", ] # Use config file if present @@ -152,27 +141,29 @@ if any(os.path.isfile(x) for x in config_locations): config = configparser.ConfigParser() config.read(config_locations) - section = config['global'] + section = config["global"] - SECRET_FILE = section.get('secret_file', SECRET_FILE) - STATIC_ROOT = section.get('static_root', STATIC_ROOT) - STATIC_URL = section.get('static_url', STATIC_URL) - MEDIA_ROOT = section.get('media_root', MEDIA_ROOT) - MEDIA_URL = section.get('media_url', MEDIA_URL) - LANGUAGE_CODE = section.get('language_code', LANGUAGE_CODE) - TIME_ZONE = section.get('time_zone', TIME_ZONE) - DEBUG = section.getboolean('debug', DEBUG) + SECRET_FILE = section.get("secret_file", SECRET_FILE) + STATIC_ROOT = section.get("static_root", STATIC_ROOT) + STATIC_URL = section.get("static_url", STATIC_URL) + MEDIA_ROOT = section.get("media_root", MEDIA_ROOT) + MEDIA_URL = section.get("media_url", MEDIA_URL) + LANGUAGE_CODE = section.get("language_code", LANGUAGE_CODE) + TIME_ZONE = section.get("time_zone", TIME_ZONE) + DEBUG = section.getboolean("debug", DEBUG) - if 'allowed_hosts' in config: - ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')] + if "allowed_hosts" in config: + ALLOWED_HOSTS = [y for x, y in config.items("allowed_hosts")] - if 'database' in config: - DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } } + if "database" in config: + DATABASES = {"default": {x.upper(): y for x, y in config.items("database")}} -ETEBASE_API_PERMISSIONS = ('rest_framework.permissions.IsAuthenticated', ) -ETEBASE_API_AUTHENTICATORS = ('django_etebase.token_auth.authentication.TokenAuthentication', - 'rest_framework.authentication.SessionAuthentication') -ETEBASE_CREATE_USER_FUNC = 'django_etebase.utils.create_user_blocked' +ETEBASE_API_PERMISSIONS = ("rest_framework.permissions.IsAuthenticated",) +ETEBASE_API_AUTHENTICATORS = ( + "django_etebase.token_auth.authentication.TokenAuthentication", + "rest_framework.authentication.SessionAuthentication", +) +ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked" # Make an `etebase_server_settings` module available to override settings. try: @@ -180,5 +171,5 @@ try: except ImportError: pass -if 'SECRET_KEY' not in locals(): +if "SECRET_KEY" not in locals(): SECRET_KEY = get_secret_from_file(SECRET_FILE) diff --git a/etebase_server/urls.py b/etebase_server/urls.py index fddc32f..f285977 100644 --- a/etebase_server/urls.py +++ b/etebase_server/urls.py @@ -5,13 +5,12 @@ from django.urls import path from django.views.generic import TemplateView urlpatterns = [ - url(r'^api/', include('django_etebase.urls')), - url(r'^admin/', admin.site.urls), - - path('', TemplateView.as_view(template_name='success.html')), + url(r"^api/", include("django_etebase.urls")), + url(r"^admin/", admin.site.urls), + path("", TemplateView.as_view(template_name="success.html")), ] if settings.DEBUG: urlpatterns += [ - url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')), + url(r"^api-auth/", include("rest_framework.urls", namespace="rest_framework")), ] diff --git a/etebase_server/utils.py b/etebase_server/utils.py index 21c99f2..64ed657 100644 --- a/etebase_server/utils.py +++ b/etebase_server/utils.py @@ -14,6 +14,7 @@ from django.core.management import utils + def get_secret_from_file(path): try: with open(path, "r") as f: diff --git a/etebase_server/wsgi.py b/etebase_server/wsgi.py index cf449a1..908f88c 100644 --- a/etebase_server/wsgi.py +++ b/etebase_server/wsgi.py @@ -11,6 +11,6 @@ import os from django.core.wsgi import get_wsgi_application -os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'etebase_server.settings') +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") application = get_wsgi_application() diff --git a/manage.py b/manage.py index b793fd2..91277fb 100755 --- a/manage.py +++ b/manage.py @@ -5,7 +5,7 @@ import sys def main(): - os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'etebase_server.settings') + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") try: from django.core.management import execute_from_command_line except ImportError as exc: @@ -17,5 +17,5 @@ def main(): execute_from_command_line(sys.argv) -if __name__ == '__main__': +if __name__ == "__main__": main() diff --git a/myauth/admin.py b/myauth/admin.py index 0ecde3f..1f4b767 100644 --- a/myauth/admin.py +++ b/myauth/admin.py @@ -6,11 +6,7 @@ from .forms import AdminUserCreationForm class UserAdmin(DjangoUserAdmin): add_form = AdminUserCreationForm - add_fieldsets = ( - (None, { - 'classes': ('wide',), - 'fields': ('username', ), - }), - ) + add_fieldsets = ((None, {"classes": ("wide",), "fields": ("username",),}),) + admin.site.register(User, UserAdmin) diff --git a/myauth/apps.py b/myauth/apps.py index 611e83d..96cb29b 100644 --- a/myauth/apps.py +++ b/myauth/apps.py @@ -2,4 +2,4 @@ from django.apps import AppConfig class MyauthConfig(AppConfig): - name = 'myauth' + name = "myauth" diff --git a/myauth/forms.py b/myauth/forms.py index 55f7299..7aacb9b 100644 --- a/myauth/forms.py +++ b/myauth/forms.py @@ -14,12 +14,12 @@ class AdminUserCreationForm(forms.ModelForm): class Meta: model = User fields = ("username",) - field_classes = {'username': UsernameField} + field_classes = {"username": UsernameField} def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) if self._meta.model.USERNAME_FIELD in self.fields: - self.fields[self._meta.model.USERNAME_FIELD].widget.attrs['autofocus'] = True + self.fields[self._meta.model.USERNAME_FIELD].widget.attrs["autofocus"] = True def save(self, commit=True): user = super().save(commit=False) @@ -27,4 +27,3 @@ class AdminUserCreationForm(forms.ModelForm): if commit: user.save() return user - diff --git a/myauth/migrations/0001_initial.py b/myauth/migrations/0001_initial.py index 1f81e95..e6c2cba 100644 --- a/myauth/migrations/0001_initial.py +++ b/myauth/migrations/0001_initial.py @@ -11,34 +11,79 @@ class Migration(migrations.Migration): initial = True dependencies = [ - ('auth', '0011_update_proxy_permissions'), + ("auth", "0011_update_proxy_permissions"), ] operations = [ migrations.CreateModel( - name='User', + name="User", fields=[ - ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), - ('password', models.CharField(max_length=128, verbose_name='password')), - ('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')), - ('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')), - ('username', models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150, unique=True, validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], verbose_name='username')), - ('first_name', models.CharField(blank=True, max_length=30, verbose_name='first name')), - ('last_name', models.CharField(blank=True, max_length=150, verbose_name='last name')), - ('email', models.EmailField(blank=True, max_length=254, verbose_name='email address')), - ('is_staff', models.BooleanField(default=False, help_text='Designates whether the user can log into this admin site.', verbose_name='staff status')), - ('is_active', models.BooleanField(default=True, help_text='Designates whether this user should be treated as active. Unselect this instead of deleting accounts.', verbose_name='active')), - ('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='date joined')), - ('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.Group', verbose_name='groups')), - ('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.Permission', verbose_name='user permissions')), - ], - options={ - 'verbose_name': 'user', - 'verbose_name_plural': 'users', - 'abstract': False, - }, - managers=[ - ('objects', django.contrib.auth.models.UserManager()), + ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")), + ("password", models.CharField(max_length=128, verbose_name="password")), + ("last_login", models.DateTimeField(blank=True, null=True, verbose_name="last login")), + ( + "is_superuser", + models.BooleanField( + default=False, + help_text="Designates that this user has all permissions without explicitly assigning them.", + verbose_name="superuser status", + ), + ), + ( + "username", + models.CharField( + error_messages={"unique": "A user with that username already exists."}, + help_text="Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.", + max_length=150, + unique=True, + validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], + verbose_name="username", + ), + ), + ("first_name", models.CharField(blank=True, max_length=30, verbose_name="first name")), + ("last_name", models.CharField(blank=True, max_length=150, verbose_name="last name")), + ("email", models.EmailField(blank=True, max_length=254, verbose_name="email address")), + ( + "is_staff", + models.BooleanField( + default=False, + help_text="Designates whether the user can log into this admin site.", + verbose_name="staff status", + ), + ), + ( + "is_active", + models.BooleanField( + default=True, + help_text="Designates whether this user should be treated as active. Unselect this instead of deleting accounts.", + verbose_name="active", + ), + ), + ("date_joined", models.DateTimeField(default=django.utils.timezone.now, verbose_name="date joined")), + ( + "groups", + models.ManyToManyField( + blank=True, + help_text="The groups this user belongs to. A user will get all permissions granted to each of their groups.", + related_name="user_set", + related_query_name="user", + to="auth.Group", + verbose_name="groups", + ), + ), + ( + "user_permissions", + models.ManyToManyField( + blank=True, + help_text="Specific permissions for this user.", + related_name="user_set", + related_query_name="user", + to="auth.Permission", + verbose_name="user permissions", + ), + ), ], + options={"verbose_name": "user", "verbose_name_plural": "users", "abstract": False,}, + managers=[("objects", django.contrib.auth.models.UserManager()),], ), ] diff --git a/myauth/migrations/0002_auto_20200515_0801.py b/myauth/migrations/0002_auto_20200515_0801.py index 3ce02b2..068c9ae 100644 --- a/myauth/migrations/0002_auto_20200515_0801.py +++ b/myauth/migrations/0002_auto_20200515_0801.py @@ -7,13 +7,20 @@ import myauth.models class Migration(migrations.Migration): dependencies = [ - ('myauth', '0001_initial'), + ("myauth", "0001_initial"), ] operations = [ migrations.AlterField( - model_name='user', - name='username', - field=models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and ./+/-/_ only.', max_length=150, unique=True, validators=[myauth.models.UnicodeUsernameValidator()], verbose_name='username'), + model_name="user", + name="username", + field=models.CharField( + error_messages={"unique": "A user with that username already exists."}, + help_text="Required. 150 characters or fewer. Letters, digits and ./+/-/_ only.", + max_length=150, + unique=True, + validators=[myauth.models.UnicodeUsernameValidator()], + verbose_name="username", + ), ), ] diff --git a/myauth/models.py b/myauth/models.py index 611555b..d6585a8 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -7,17 +7,14 @@ from django.utils.translation import gettext_lazy as _ @deconstructible class UnicodeUsernameValidator(validators.RegexValidator): - regex = r'^[\w.-]+\Z' - message = _( - 'Enter a valid username. This value may contain only letters, ' - 'numbers, and ./-/_ characters.' - ) + regex = r"^[\w.-]+\Z" + message = _("Enter a valid username. This value may contain only letters, " "numbers, and ./-/_ characters.") flags = 0 class UserManager(DjangoUserManager): def get_by_natural_key(self, username): - return self.get(**{self.model.USERNAME_FIELD + '__iexact': username}) + return self.get(**{self.model.USERNAME_FIELD + "__iexact": username}) class User(AbstractUser): @@ -26,14 +23,12 @@ class User(AbstractUser): objects = UserManager() username = models.CharField( - _('username'), + _("username"), max_length=150, unique=True, - help_text=_('Required. 150 characters or fewer. Letters, digits and ./-/_ only.'), + help_text=_("Required. 150 characters or fewer. Letters, digits and ./-/_ only."), validators=[username_validator], - error_messages={ - 'unique': _("A user with that username already exists."), - }, + error_messages={"unique": _("A user with that username already exists."),}, ) @classmethod diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..e34796e --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,2 @@ +[tool.black] +line-length = 120 \ No newline at end of file diff --git a/requirements.in/development.txt b/requirements.in/development.txt index c752bfb..a956471 100644 --- a/requirements.in/development.txt +++ b/requirements.in/development.txt @@ -1,3 +1,4 @@ coverage pip-tools pywatchman +black \ No newline at end of file From b6919d17beb92ce019ca6a9f223282307f3df468 Mon Sep 17 00:00:00 2001 From: Michael Nahkies Date: Sat, 14 Nov 2020 15:16:13 +0000 Subject: [PATCH 322/511] chore: fix broken links in README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ebb384e..dc05b31 100644 --- a/README.md +++ b/README.md @@ -34,9 +34,9 @@ pip install -r requirements.txt # Configuration -If you are familiar with Django you can just edit the [settings file](etesync_server/settings.py) +If you are familiar with Django you can just edit the [settings file](etebase_server/settings.py) according to the [Django deployment checklist](https://docs.djangoproject.com/en/dev/howto/deployment/checklist/). -If you are not, we also provide a simple [configuration file](https://github.com/etesync/server/blob/etebase/etebase-server.ini.example) for easy deployment which you can use. +If you are not, we also provide a simple [configuration file](etebase-server.ini.example) for easy deployment which you can use. To use the easy configuration file rename it to `etebase-server.ini` and place it either at the root of this repository or in `/etc/etebase-server`. There is also a [wikipage](https://github.com/etesync/server/wiki/Basic-Setup-Etebase-(EteSync-v2)) detailing this basic setup. From 4f4bdc7b6bfc3c5b22b1456366c5ec87d24a6688 Mon Sep 17 00:00:00 2001 From: Michael Nahkies Date: Sun, 15 Nov 2020 09:55:53 +0000 Subject: [PATCH 323/511] chore: fix install instructions in readme, .gitignore venv / secret.txt (#68) --- .gitignore | 3 ++- README.md | 5 ++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index 7f220af..2c3f6a9 100644 --- a/.gitignore +++ b/.gitignore @@ -7,9 +7,10 @@ Session.vim /.coverage /tmp /media +/.idea __pycache__ .*.swp - /etebase_server_settings.py +/secret.txt diff --git a/README.md b/README.md index dc05b31..9940271 100644 --- a/README.md +++ b/README.md @@ -23,11 +23,10 @@ Then just clone the git repo and set up this app: git clone https://github.com/etesync/server.git etebase cd etebase -git checkout etebase # Set up the environment and deps -virtualenv -p python3 venv # If doesn't work, try: virtualenv3 venv -source venv/bin/activate +virtualenv -p python3 .venv # If doesn't work, try: virtualenv3 .venv +source .venv/bin/activate pip install -r requirements.txt ``` From 13a46cb1dbf44326b3cd8c22a37b170e5894424c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 19 Nov 2020 10:12:12 +0200 Subject: [PATCH 324/511] Myauth: add missing migration. Fixes #70 --- myauth/migrations/0003_auto_20201119_0810.py | 37 ++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 myauth/migrations/0003_auto_20201119_0810.py diff --git a/myauth/migrations/0003_auto_20201119_0810.py b/myauth/migrations/0003_auto_20201119_0810.py new file mode 100644 index 0000000..cfd2ec3 --- /dev/null +++ b/myauth/migrations/0003_auto_20201119_0810.py @@ -0,0 +1,37 @@ +# Generated by Django 3.1.1 on 2020-11-19 08:10 + +from django.db import migrations, models +import myauth.models + + +class Migration(migrations.Migration): + + dependencies = [ + ("myauth", "0002_auto_20200515_0801"), + ] + + operations = [ + migrations.AlterModelManagers( + name="user", + managers=[ + ("objects", myauth.models.UserManager()), + ], + ), + migrations.AlterField( + model_name="user", + name="first_name", + field=models.CharField(blank=True, max_length=150, verbose_name="first name"), + ), + migrations.AlterField( + model_name="user", + name="username", + field=models.CharField( + error_messages={"unique": "A user with that username already exists."}, + help_text="Required. 150 characters or fewer. Letters, digits and ./-/_ only.", + max_length=150, + unique=True, + validators=[myauth.models.UnicodeUsernameValidator()], + verbose_name="username", + ), + ), + ] From 1c8f7cdc609065c9e1700c8f5f159f65f4558cec Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 19 Nov 2020 10:12:42 +0200 Subject: [PATCH 325/511] Update changelog. --- ChangeLog.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index d3adef0..3de4eef 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,8 @@ # Changelog +## Version 0.5.3 +* Add missing migration + ## Version 0.5.2 * Fix issues with host verification failing with a custom port * Add env variable to change configuration file path. From 49da4ea6660a3bbcc0afa3764ca8f083f75afa84 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 19 Nov 2020 14:42:57 +0200 Subject: [PATCH 326/511] README: document MEDIA_ROOT and add a section about backups --- README.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9940271..f347133 100644 --- a/README.md +++ b/README.md @@ -41,12 +41,14 @@ To use the easy configuration file rename it to `etebase-server.ini` and place i There is also a [wikipage](https://github.com/etesync/server/wiki/Basic-Setup-Etebase-(EteSync-v2)) detailing this basic setup. Some particular settings that should be edited are: - * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) + * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-ALLOWED_HOSTS) -- this is the list of host/domain names or addresses on which the app will be served - * [`DEBUG`](https://docs.djangoproject.com/en/1.11/ref/settings/#debug) + * [`DEBUG`](https://docs.djangoproject.com/en/dev/ref/settings/#debug) -- handy for debugging, set to `False` for production - * [`SECRET_KEY`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-SECRET_KEY) + * [`MEDIA_ROOT`](https://docs.djangoproject.com/en/dev/ref/settings/#media-root) + -- the path to the directory that will hold user data. + * [`SECRET_KEY`](https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-SECRET_KEY) -- an ephemeral secret used for various cryptographic signing and token generation purposes. See below for how default configuration of `SECRET_KEY` works for this project. @@ -76,6 +78,12 @@ Etebase is based on Django so you should refer to one of the following The webserver should also be configured to serve Etebase using TLS. A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-Etebase) as well. +# Data locations and backups + +The server stores user data in two different locations that need to be backed up: +1. The database - how to backup depends on which database you use. +2. The `MEDIA_ROOT` - the path where user data is stored. + # Usage Create yourself an admin user: From d893d35c6f792b06888cdf990937e83f01eb14e5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 20 Nov 2020 18:11:35 +0200 Subject: [PATCH 327/511] Fix the host checks to only check against hostname. Fixes https://github.com/etesync/etesync-web/issues/183 As discussed in #66 Continuation of 843b59a0ac1e2076a514b9c52fc5c6941b27dff0. --- django_etebase/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 7dd7526..f3abcbc 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -708,7 +708,7 @@ class AuthenticationViewSet(viewsets.ViewSet): elif challenge_data["userId"] != user.id: content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host.split(":", 1)[0] != request.get_host(): + elif not settings.DEBUG and host.split(':', 1)[0] != request.get_host().split(':', 1)[0]: detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) content = {"code": "wrong_host", "detail": detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) From 5792cd5418283d2ad590e8ef1bcdbbe601ed7365 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 27 Nov 2020 10:29:24 +0200 Subject: [PATCH 328/511] README: add a TL;DR for licensing Added it following questions via email. --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index f347133..12f7747 100644 --- a/README.md +++ b/README.md @@ -136,6 +136,16 @@ Here are the update steps: 4. Run the migration tool to migrate all of your data. 5. Add your new EteSync 2.0 accounts to all of your devices. +# License + +Etebase is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation. See the [LICENSE](./LICENSE) for more information. + +A quick summary can be found [on tldrlegal](https://tldrlegal.com/license/gnu-affero-general-public-license-v3-(agpl-3.0)). Though in even simpler terms (not part of the license, and not legal advice): you can use it in however way you want, including self-hosting and commercial offerings as long as you release the code to any modifications you have made to the server software (clients are not affected). + +## Commercial licensing + +For commercial licensing options, contact license@etebase.com + # Supporting Etebase Please consider registering an account even if you self-host in order to support the development of Etebase, or visit the [contribution](https://www.etesync.com/contribute/) for more information on how to support the service. From bb4a8c998e147c5c55e6c75a15cd8acac70820fb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 29 Nov 2020 17:02:49 +0200 Subject: [PATCH 329/511] README: updateinformation about passing the Host header As mentioned in https://github.com/etesync/server/issues/75#issuecomment-735370709 --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 12f7747..cfb0be9 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ There is also a [wikipage](https://github.com/etesync/server/wiki/Basic-Setup-Et Some particular settings that should be edited are: * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-ALLOWED_HOSTS) -- this is the list of host/domain names or addresses on which the app -will be served +will be served. For example: `etebase.example.com` * [`DEBUG`](https://docs.djangoproject.com/en/dev/ref/settings/#debug) -- handy for debugging, set to `False` for production * [`MEDIA_ROOT`](https://docs.djangoproject.com/en/dev/ref/settings/#media-root) @@ -78,6 +78,8 @@ Etebase is based on Django so you should refer to one of the following The webserver should also be configured to serve Etebase using TLS. A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-Etebase) as well. +The Etebase server needs to be aware of the URL it's been served as, so make sure to forward the `Host` header to the server if using a reverse proxy. For example, you would need to use the following directive in nginx: `proxy_set_header Host $host;`. + # Data locations and backups The server stores user data in two different locations that need to be backed up: From 7513b058ba51ec911c4ba6a0f7851889b461b055 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 1 Dec 2020 10:08:04 +0200 Subject: [PATCH 330/511] FUNDING.yml: add funding links --- .github/FUNDING.yml | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .github/FUNDING.yml diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..3ece078 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,2 @@ +github: etesync +custom: https://www.etesync.com/contribute/#donate From c790b5f4899fbb6f17567e6ed9053b741f227300 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 1 Dec 2020 12:45:23 +0200 Subject: [PATCH 331/511] Reformat some files using black. --- django_etebase/exceptions.py | 5 +++- django_etebase/migrations/0001_initial.py | 20 ++++++++++++---- .../migrations/0003_collectioninvitation.py | 4 +++- .../0004_collectioninvitation_version.py | 4 +++- .../migrations/0005_auto_20200526_1021.py | 6 ++++- .../0013_collectionmemberremoved.py | 4 +++- .../migrations/0014_auto_20200602_1558.py | 6 ++++- .../0015_collectionitemrevision_salt.py | 4 +++- .../migrations/0016_auto_20200623_0820.py | 15 +++++++++--- ...0020_remove_collectionitemrevision_salt.py | 5 +++- .../migrations/0022_auto_20200804_1059.py | 5 +++- .../migrations/0025_auto_20200804_1216.py | 10 ++++++-- .../migrations/0026_auto_20200907_0752.py | 12 ++++++++-- .../migrations/0029_auto_20200907_0801.py | 10 ++++++-- django_etebase/models.py | 16 ++++++++++--- django_etebase/serializers.py | 24 +++++++++++++++---- django_etebase/views.py | 7 ++++-- 17 files changed, 125 insertions(+), 32 deletions(-) diff --git a/django_etebase/exceptions.py b/django_etebase/exceptions.py index f3aa08a..437a71c 100644 --- a/django_etebase/exceptions.py +++ b/django_etebase/exceptions.py @@ -4,6 +4,9 @@ from rest_framework import serializers, status class EtebaseValidationError(serializers.ValidationError): def __init__(self, code, detail, status_code=status.HTTP_400_BAD_REQUEST): super().__init__( - {"code": code, "detail": detail,} + { + "code": code, + "detail": detail, + } ) self.status_code = status_code diff --git a/django_etebase/migrations/0001_initial.py b/django_etebase/migrations/0001_initial.py index 86f0fa6..42ee022 100644 --- a/django_etebase/migrations/0001_initial.py +++ b/django_etebase/migrations/0001_initial.py @@ -33,7 +33,9 @@ class Migration(migrations.Migration): ("version", models.PositiveSmallIntegerField()), ("owner", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={"unique_together": {("uid", "owner")},}, + options={ + "unique_together": {("uid", "owner")}, + }, ), migrations.CreateModel( name="CollectionItem", @@ -61,7 +63,9 @@ class Migration(migrations.Migration): ), ), ], - options={"unique_together": {("uid", "collection")},}, + options={ + "unique_together": {("uid", "collection")}, + }, ), migrations.CreateModel( name="CollectionItemChunk", @@ -122,7 +126,9 @@ class Migration(migrations.Migration): ), ), ], - options={"unique_together": {("item", "current")},}, + options={ + "unique_together": {("item", "current")}, + }, ), migrations.CreateModel( name="RevisionChunkRelation", @@ -145,7 +151,9 @@ class Migration(migrations.Migration): ), ), ], - options={"ordering": ("id",),}, + options={ + "ordering": ("id",), + }, ), migrations.CreateModel( name="CollectionMember", @@ -170,6 +178,8 @@ class Migration(migrations.Migration): ), ("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={"unique_together": {("user", "collection")},}, + options={ + "unique_together": {("user", "collection")}, + }, ), ] diff --git a/django_etebase/migrations/0003_collectioninvitation.py b/django_etebase/migrations/0003_collectioninvitation.py index 1b416ab..4aa05c9 100644 --- a/django_etebase/migrations/0003_collectioninvitation.py +++ b/django_etebase/migrations/0003_collectioninvitation.py @@ -54,6 +54,8 @@ class Migration(migrations.Migration): ), ), ], - options={"unique_together": {("user", "fromMember")},}, + options={ + "unique_together": {("user", "fromMember")}, + }, ), ] diff --git a/django_etebase/migrations/0004_collectioninvitation_version.py b/django_etebase/migrations/0004_collectioninvitation_version.py index 29ae3f1..40a290e 100644 --- a/django_etebase/migrations/0004_collectioninvitation_version.py +++ b/django_etebase/migrations/0004_collectioninvitation_version.py @@ -11,6 +11,8 @@ class Migration(migrations.Migration): operations = [ migrations.AddField( - model_name="collectioninvitation", name="version", field=models.PositiveSmallIntegerField(default=1), + model_name="collectioninvitation", + name="version", + field=models.PositiveSmallIntegerField(default=1), ), ] diff --git a/django_etebase/migrations/0005_auto_20200526_1021.py b/django_etebase/migrations/0005_auto_20200526_1021.py index 3775277..9855ff1 100644 --- a/django_etebase/migrations/0005_auto_20200526_1021.py +++ b/django_etebase/migrations/0005_auto_20200526_1021.py @@ -10,5 +10,9 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RenameField(model_name="userinfo", old_name="pubkey", new_name="loginPubkey",), + migrations.RenameField( + model_name="userinfo", + old_name="pubkey", + new_name="loginPubkey", + ), ] diff --git a/django_etebase/migrations/0013_collectionmemberremoved.py b/django_etebase/migrations/0013_collectionmemberremoved.py index 4481e80..d8c57ea 100644 --- a/django_etebase/migrations/0013_collectionmemberremoved.py +++ b/django_etebase/migrations/0013_collectionmemberremoved.py @@ -33,6 +33,8 @@ class Migration(migrations.Migration): ), ("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={"unique_together": {("user", "collection")},}, + options={ + "unique_together": {("user", "collection")}, + }, ), ] diff --git a/django_etebase/migrations/0014_auto_20200602_1558.py b/django_etebase/migrations/0014_auto_20200602_1558.py index 42bed52..ec0de46 100644 --- a/django_etebase/migrations/0014_auto_20200602_1558.py +++ b/django_etebase/migrations/0014_auto_20200602_1558.py @@ -10,5 +10,9 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RenameField(model_name="userinfo", old_name="encryptedSeckey", new_name="encryptedContent",), + migrations.RenameField( + model_name="userinfo", + old_name="encryptedSeckey", + new_name="encryptedContent", + ), ] diff --git a/django_etebase/migrations/0015_collectionitemrevision_salt.py b/django_etebase/migrations/0015_collectionitemrevision_salt.py index c4dc3e9..3a13baa 100644 --- a/django_etebase/migrations/0015_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0015_collectionitemrevision_salt.py @@ -11,6 +11,8 @@ class Migration(migrations.Migration): operations = [ migrations.AddField( - model_name="collectionitemrevision", name="salt", field=models.BinaryField(default=b"", editable=True), + model_name="collectionitemrevision", + name="salt", + field=models.BinaryField(default=b"", editable=True), ), ] diff --git a/django_etebase/migrations/0016_auto_20200623_0820.py b/django_etebase/migrations/0016_auto_20200623_0820.py index a273b0d..14871a2 100644 --- a/django_etebase/migrations/0016_auto_20200623_0820.py +++ b/django_etebase/migrations/0016_auto_20200623_0820.py @@ -21,7 +21,16 @@ class Migration(migrations.Migration): to="django_etebase.CollectionItem", ), ), - migrations.AlterUniqueTogether(name="collection", unique_together=set(),), - migrations.RemoveField(model_name="collection", name="uid",), - migrations.RemoveField(model_name="collection", name="version",), + migrations.AlterUniqueTogether( + name="collection", + unique_together=set(), + ), + migrations.RemoveField( + model_name="collection", + name="uid", + ), + migrations.RemoveField( + model_name="collection", + name="version", + ), ] diff --git a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py index 21d0337..1b69a5c 100644 --- a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py @@ -10,5 +10,8 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RemoveField(model_name="collectionitemrevision", name="salt",), + migrations.RemoveField( + model_name="collectionitemrevision", + name="salt", + ), ] diff --git a/django_etebase/migrations/0022_auto_20200804_1059.py b/django_etebase/migrations/0022_auto_20200804_1059.py index 60af33f..bc4bad9 100644 --- a/django_etebase/migrations/0022_auto_20200804_1059.py +++ b/django_etebase/migrations/0022_auto_20200804_1059.py @@ -10,5 +10,8 @@ class Migration(migrations.Migration): ] operations = [ - migrations.AlterUniqueTogether(name="collectionitemchunk", unique_together={("item", "uid")},), + migrations.AlterUniqueTogether( + name="collectionitemchunk", + unique_together={("item", "uid")}, + ), ] diff --git a/django_etebase/migrations/0025_auto_20200804_1216.py b/django_etebase/migrations/0025_auto_20200804_1216.py index 91bf4c8..995d275 100644 --- a/django_etebase/migrations/0025_auto_20200804_1216.py +++ b/django_etebase/migrations/0025_auto_20200804_1216.py @@ -18,6 +18,12 @@ class Migration(migrations.Migration): on_delete=django.db.models.deletion.CASCADE, related_name="chunks", to="django_etebase.Collection" ), ), - migrations.AlterUniqueTogether(name="collectionitemchunk", unique_together={("collection", "uid")},), - migrations.RemoveField(model_name="collectionitemchunk", name="item",), + migrations.AlterUniqueTogether( + name="collectionitemchunk", + unique_together={("collection", "uid")}, + ), + migrations.RemoveField( + model_name="collectionitemchunk", + name="item", + ), ] diff --git a/django_etebase/migrations/0026_auto_20200907_0752.py b/django_etebase/migrations/0026_auto_20200907_0752.py index 3283654..1c90610 100644 --- a/django_etebase/migrations/0026_auto_20200907_0752.py +++ b/django_etebase/migrations/0026_auto_20200907_0752.py @@ -10,6 +10,14 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RenameField(model_name="collectioninvitation", old_name="accessLevel", new_name="accessLevelOld",), - migrations.RenameField(model_name="collectionmember", old_name="accessLevel", new_name="accessLevelOld",), + migrations.RenameField( + model_name="collectioninvitation", + old_name="accessLevel", + new_name="accessLevelOld", + ), + migrations.RenameField( + model_name="collectionmember", + old_name="accessLevel", + new_name="accessLevelOld", + ), ] diff --git a/django_etebase/migrations/0029_auto_20200907_0801.py b/django_etebase/migrations/0029_auto_20200907_0801.py index f3bfe61..1e00ffe 100644 --- a/django_etebase/migrations/0029_auto_20200907_0801.py +++ b/django_etebase/migrations/0029_auto_20200907_0801.py @@ -10,6 +10,12 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RemoveField(model_name="collectioninvitation", name="accessLevelOld",), - migrations.RemoveField(model_name="collectionmember", name="accessLevelOld",), + migrations.RemoveField( + model_name="collectioninvitation", + name="accessLevelOld", + ), + migrations.RemoveField( + model_name="collectionmember", + name="accessLevelOld", + ), ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 691947d..00bbf71 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -182,7 +182,10 @@ class CollectionMember(models.Model): user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) collectionType = models.ForeignKey(CollectionType, on_delete=models.PROTECT, null=True) - accessLevel = models.IntegerField(choices=AccessLevels.choices, default=AccessLevels.READ_ONLY,) + accessLevel = models.IntegerField( + choices=AccessLevels.choices, + default=AccessLevels.READ_ONLY, + ) class Meta: unique_together = ("user", "collection") @@ -193,7 +196,11 @@ class CollectionMember(models.Model): def revoke(self): with transaction.atomic(): CollectionMemberRemoved.objects.update_or_create( - collection=self.collection, user=self.user, defaults={"stoken": Stoken.objects.create(),}, + collection=self.collection, + user=self.user, + defaults={ + "stoken": Stoken.objects.create(), + }, ) self.delete() @@ -220,7 +227,10 @@ class CollectionInvitation(models.Model): user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name="incoming_invitations", on_delete=models.CASCADE) signedEncryptionKey = models.BinaryField(editable=False, blank=False, null=False) - accessLevel = models.IntegerField(choices=AccessLevels.choices, default=AccessLevels.READ_ONLY,) + accessLevel = models.IntegerField( + choices=AccessLevels.choices, + default=AccessLevels.READ_ONLY, + ) class Meta: unique_together = ("user", "fromMember") diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index ef3b296..b0769ef 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -156,7 +156,11 @@ class BetterErrorsMixin: else: message = str(error) ret.append( - {"field": field_name, "code": error.code, "detail": message,} + { + "field": field_name, + "code": error.code, + "detail": message, + } ) return ret @@ -169,7 +173,11 @@ class BetterErrorsMixin: raise EtebaseValidationError(err.code, err.message) raise serializers.ValidationError( - {"code": "field_errors", "detail": "Field validations failed.", "errors": errors,} + { + "code": "field_errors", + "detail": "Field validations failed.", + "errors": errors, + } ) @@ -343,7 +351,11 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField(source="user", read_only=True, style={"base_template": "input.html"},) + username = UserSlugRelatedField( + source="user", + read_only=True, + style={"base_template": "input.html"}, + ) class Meta: model = models.CollectionMember @@ -365,7 +377,11 @@ class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer) class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField(source="user", queryset=User.objects, style={"base_template": "input.html"},) + username = UserSlugRelatedField( + source="user", + queryset=User.objects, + style={"base_template": "input.html"}, + ) collection = serializers.CharField(source="collection.uid") fromUsername = BinaryBase64Field(source="fromMember.user.username", read_only=True) fromPubkey = BinaryBase64Field(source="fromMember.user.userinfo.pubkey", read_only=True) diff --git a/django_etebase/views.py b/django_etebase/views.py index f3abcbc..8c05948 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -413,7 +413,10 @@ class CollectionItemViewSet(BaseViewSet): return Response(ret, status=status.HTTP_200_OK) return Response( - {"items": serializer.errors, "deps": deps_serializer.errors if deps is not None else [],}, + { + "items": serializer.errors, + "deps": deps_serializer.errors if deps is not None else [], + }, status=status.HTTP_409_CONFLICT, ) @@ -708,7 +711,7 @@ class AuthenticationViewSet(viewsets.ViewSet): elif challenge_data["userId"] != user.id: content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host.split(':', 1)[0] != request.get_host().split(':', 1)[0]: + elif not settings.DEBUG and host.split(":", 1)[0] != request.get_host().split(":", 1)[0]: detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) content = {"code": "wrong_host", "detail": detail} return Response(content, status=status.HTTP_400_BAD_REQUEST) From bb070639a3abfcd2ad39466833cc71abf8db4599 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 4 Dec 2020 18:19:54 +0200 Subject: [PATCH 332/511] Collection: fix the slow performance when calculating stoken. We were running a very expensive query instead of the much simpler and more efficient alternative we just introduced. --- django_etebase/models.py | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index 00bbf71..b8ead94 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -17,7 +17,7 @@ from pathlib import Path from django.db import models, transaction from django.conf import settings from django.core.validators import RegexValidator -from django.db.models import Q +from django.db.models import Max from django.utils.functional import cached_property from django.utils.crypto import get_random_string @@ -56,18 +56,13 @@ class Collection(models.Model): @cached_property def stoken(self): - stoken = ( - Stoken.objects.filter( - Q(collectionitemrevision__item__collection=self) | Q(collectionmember__collection=self) - ) - .order_by("id") - .last() - ) - - if stoken is None: + stoken1 = self.items.aggregate(stoken=Max("revisions__stoken"))["stoken"] or 0 + stoken2 = self.members.aggregate(stoken=Max("stoken"))["stoken"] or 0 + stoken_id = max(stoken1, stoken2) + if stoken_id == 0: raise Exception("stoken is None. Should never happen") - return stoken.uid + return Stoken.objects.get(id=stoken_id).uid def validate_unique(self, exclude=None): super().validate_unique(exclude=exclude) From 4ce96e043e55070e7a0eed98592c545dc435c2d4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 4 Dec 2020 18:55:22 +0200 Subject: [PATCH 333/511] Collection: further improve stoken performance. We merged the two queries into one and we made it like in the view, so we can now merge the two instead of having two implementations. --- django_etebase/models.py | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index b8ead94..d1095fb 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -17,7 +17,8 @@ from pathlib import Path from django.db import models, transaction from django.conf import settings from django.core.validators import RegexValidator -from django.db.models import Max +from django.db.models import Max, Value as V +from django.db.models.functions import Coalesce, Greatest from django.utils.functional import cached_property from django.utils.crypto import get_random_string @@ -39,6 +40,8 @@ class Collection(models.Model): main_item = models.OneToOneField("CollectionItem", related_name="parent", null=True, on_delete=models.SET_NULL) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + stoken_id_fields = ["items__revisions__stoken", "members__stoken"] + def __str__(self): return self.uid @@ -56,9 +59,15 @@ class Collection(models.Model): @cached_property def stoken(self): - stoken1 = self.items.aggregate(stoken=Max("revisions__stoken"))["stoken"] or 0 - stoken2 = self.members.aggregate(stoken=Max("stoken"))["stoken"] or 0 - stoken_id = max(stoken1, stoken2) + aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] + max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] + stoken_id = ( + self.__class__.objects.filter(main_item=self.main_item) + .annotate(max_stoken=max_stoken) + .values("max_stoken") + .first()["max_stoken"] + ) + if stoken_id == 0: raise Exception("stoken is None. Should never happen") From 2d0bcbdc2014ed6be428082bbb37e6a822a8a1a8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 4 Dec 2020 19:00:14 +0200 Subject: [PATCH 334/511] Stoken annotation: move it all to one place to reduce duplication. --- django_etebase/models.py | 15 +++++++++++---- django_etebase/views.py | 15 ++++++--------- 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index d1095fb..af4d022 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -31,6 +31,11 @@ from .exceptions import EtebaseValidationError UidValidator = RegexValidator(regex=r"^[a-zA-Z0-9\-_]{20,}$", message="Not a valid UID") +def stoken_annotation_builder(stoken_id_fields): + aggr_fields = [Coalesce(Max(field), V(0)) for field in stoken_id_fields] + return Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] + + class CollectionType(models.Model): owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True) @@ -40,7 +45,7 @@ class Collection(models.Model): main_item = models.OneToOneField("CollectionItem", related_name="parent", null=True, on_delete=models.SET_NULL) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) - stoken_id_fields = ["items__revisions__stoken", "members__stoken"] + stoken_annotation = stoken_annotation_builder(["items__revisions__stoken", "members__stoken"]) def __str__(self): return self.uid @@ -59,11 +64,9 @@ class Collection(models.Model): @cached_property def stoken(self): - aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] - max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] stoken_id = ( self.__class__.objects.filter(main_item=self.main_item) - .annotate(max_stoken=max_stoken) + .annotate(max_stoken=self.stoken_annotation) .values("max_stoken") .first()["max_stoken"] ) @@ -94,6 +97,8 @@ class CollectionItem(models.Model): version = models.PositiveSmallIntegerField() encryptionKey = models.BinaryField(editable=True, blank=False, null=True) + stoken_annotation = stoken_annotation_builder(["revisions__stoken"]) + class Meta: unique_together = ("uid", "collection") @@ -191,6 +196,8 @@ class CollectionMember(models.Model): default=AccessLevels.READ_ONLY, ) + stoken_annotation = stoken_annotation_builder(["stoken"]) + class Meta: unique_together = ("user", "collection") diff --git a/django_etebase/views.py b/django_etebase/views.py index 8c05948..d1266c6 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -18,8 +18,7 @@ from django.conf import settings from django.contrib.auth import get_user_model, user_logged_in, user_logged_out from django.core.exceptions import PermissionDenied from django.db import transaction, IntegrityError -from django.db.models import Max, Value as V, Q -from django.db.models.functions import Coalesce, Greatest +from django.db.models import Q from django.http import HttpResponseBadRequest, HttpResponse, Http404 from django.shortcuts import get_object_or_404 @@ -94,7 +93,7 @@ class BaseViewSet(viewsets.ModelViewSet): permission_classes = tuple(app_settings.API_PERMISSIONS) renderer_classes = [JSONRenderer, MessagePackRenderer] + ([BrowsableAPIRenderer] if settings.DEBUG else []) parser_classes = [JSONParser, MessagePackParser, FormParser, MultiPartParser] - stoken_id_fields = None + stoken_annotation = None def get_serializer_class(self): serializer_class = self.serializer_class @@ -125,9 +124,7 @@ class BaseViewSet(viewsets.ModelViewSet): def filter_by_stoken(self, request, queryset): stoken_rev = self.get_stoken_obj(request) - aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] - max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] - queryset = queryset.annotate(max_stoken=max_stoken).order_by("max_stoken") + queryset = queryset.annotate(max_stoken=self.stoken_annotation).order_by("max_stoken") if stoken_rev is not None: queryset = queryset.filter(max_stoken__gt=stoken_rev.id) @@ -179,7 +176,7 @@ class CollectionViewSet(BaseViewSet): serializer_class = CollectionSerializer lookup_field = "main_item__uid" lookup_url_kwarg = "uid" - stoken_id_fields = ["items__revisions__stoken__id", "members__stoken__id"] + stoken_annotation = Collection.stoken_annotation def get_queryset(self, queryset=None): if queryset is None: @@ -262,7 +259,7 @@ class CollectionItemViewSet(BaseViewSet): queryset = CollectionItem.objects.all() serializer_class = CollectionItemSerializer lookup_field = "uid" - stoken_id_fields = ["revisions__stoken__id"] + stoken_annotation = CollectionItem.stoken_annotation def get_queryset(self): collection_uid = self.kwargs["collection_uid"] @@ -482,7 +479,7 @@ class CollectionMemberViewSet(BaseViewSet): serializer_class = CollectionMemberSerializer lookup_field = f"user__{User.USERNAME_FIELD}__iexact" lookup_url_kwarg = "username" - stoken_id_fields = ["stoken__id"] + stoken_annotation = CollectionMember.stoken_annotation # FIXME: need to make sure that there's always an admin, and maybe also don't let an owner remove adm access # (if we want to transfer, we need to do that specifically) From 057b908565072bf9b1003410c0080f42294d446a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 4 Dec 2020 19:15:10 +0200 Subject: [PATCH 335/511] Update changelog. --- ChangeLog.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index 3de4eef..b47be77 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,9 @@ # Changelog +## Version 0.6.0 +* Fix stoken calculation performance - was VERY slow in some rare cases +* Fix issues with host verification failing with a custom port - part 2 + ## Version 0.5.3 * Add missing migration From baa42d040d1012267107866ad4286292063e6991 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 14 Dec 2020 13:30:30 +0200 Subject: [PATCH 336/511] Collection: also save the collection UID on the model itself. This enables us to have db-constraints for making sure that UIDs are unique, as well as being more efficient for lookups (which are very common). The UID should always be the same as the main_item.uid, though that's easily enforced as neither of them is allowed to change. --- .../migrations/0033_collection_uid.py | 19 ++++++++++++++++ .../migrations/0034_auto_20201214_1124.py | 22 +++++++++++++++++++ .../migrations/0035_auto_20201214_1126.py | 19 ++++++++++++++++ .../migrations/0036_auto_20201214_1128.py | 19 ++++++++++++++++ django_etebase/models.py | 20 ++--------------- django_etebase/serializers.py | 4 ++-- django_etebase/views.py | 20 ++++++++--------- 7 files changed, 93 insertions(+), 30 deletions(-) create mode 100644 django_etebase/migrations/0033_collection_uid.py create mode 100644 django_etebase/migrations/0034_auto_20201214_1124.py create mode 100644 django_etebase/migrations/0035_auto_20201214_1126.py create mode 100644 django_etebase/migrations/0036_auto_20201214_1128.py diff --git a/django_etebase/migrations/0033_collection_uid.py b/django_etebase/migrations/0033_collection_uid.py new file mode 100644 index 0000000..e4a829e --- /dev/null +++ b/django_etebase/migrations/0033_collection_uid.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:21 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0032_auto_20201013_1409'), + ] + + operations = [ + migrations.AddField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/migrations/0034_auto_20201214_1124.py b/django_etebase/migrations/0034_auto_20201214_1124.py new file mode 100644 index 0000000..b05060b --- /dev/null +++ b/django_etebase/migrations/0034_auto_20201214_1124.py @@ -0,0 +1,22 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:24 + +from django.db import migrations + + +def update_collection_uid(apps, schema_editor): + Collection = apps.get_model("django_etebase", "Collection") + + for collection in Collection.objects.all(): + collection.uid = collection.main_item.uid + collection.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ("django_etebase", "0033_collection_uid"), + ] + + operations = [ + migrations.RunPython(update_collection_uid), + ] diff --git a/django_etebase/migrations/0035_auto_20201214_1126.py b/django_etebase/migrations/0035_auto_20201214_1126.py new file mode 100644 index 0000000..489c5e0 --- /dev/null +++ b/django_etebase/migrations/0035_auto_20201214_1126.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:26 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0034_auto_20201214_1124'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/migrations/0036_auto_20201214_1128.py b/django_etebase/migrations/0036_auto_20201214_1128.py new file mode 100644 index 0000000..241adf1 --- /dev/null +++ b/django_etebase/migrations/0036_auto_20201214_1128.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:28 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0035_auto_20201214_1126'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index af4d022..11b6506 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -43,6 +43,8 @@ class CollectionType(models.Model): class Collection(models.Model): main_item = models.OneToOneField("CollectionItem", related_name="parent", null=True, on_delete=models.SET_NULL) + # The same as main_item.uid, we just also save it here so we have DB constraints for uniqueness (and efficiency) + uid = models.CharField(db_index=True, unique=True, blank=False, max_length=43, validators=[UidValidator]) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) stoken_annotation = stoken_annotation_builder(["items__revisions__stoken", "members__stoken"]) @@ -50,10 +52,6 @@ class Collection(models.Model): def __str__(self): return self.uid - @cached_property - def uid(self): - return self.main_item.uid - @property def content(self): return self.main_item.content @@ -76,20 +74,6 @@ class Collection(models.Model): return Stoken.objects.get(id=stoken_id).uid - def validate_unique(self, exclude=None): - super().validate_unique(exclude=exclude) - if exclude is None or "main_item" in exclude: - return - - if ( - self.__class__.objects.filter(owner=self.owner, main_item__uid=self.main_item.uid) - .exclude(id=self.id) - .exists() - ): - raise EtebaseValidationError( - "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT - ) - class CollectionItem(models.Model): uid = models.CharField(db_index=True, blank=False, max_length=43, validators=[UidValidator]) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index b0769ef..9628986 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -313,13 +313,13 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): user = validated_data.get("owner") main_item_data = validated_data.pop("main_item") + uid = main_item_data.get("uid") etag = main_item_data.pop("etag") revision_data = main_item_data.pop("content") - instance = self.__class__.Meta.model(**validated_data) + instance = self.__class__.Meta.model(uid=uid, **validated_data) with transaction.atomic(): - _ = self.__class__.Meta.model.objects.select_for_update().filter(owner=user) if etag is not None: raise EtebaseValidationError("bad_etag", "etag is not null") diff --git a/django_etebase/views.py b/django_etebase/views.py index d1266c6..44c3383 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -174,7 +174,7 @@ class CollectionViewSet(BaseViewSet): permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly,) queryset = Collection.objects.all() serializer_class = CollectionSerializer - lookup_field = "main_item__uid" + lookup_field = "uid" lookup_url_kwarg = "uid" stoken_annotation = Collection.stoken_annotation @@ -246,7 +246,7 @@ class CollectionViewSet(BaseViewSet): # can point to the most recent collection change rather than most recent removed membership. remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) - remed = remed_qs.values_list("collection__main_item__uid", flat=True) + remed = remed_qs.values_list("collection__uid", flat=True) if len(remed) > 0: ret["removedMemberships"] = [{"uid": x} for x in remed] @@ -264,7 +264,7 @@ class CollectionItemViewSet(BaseViewSet): def get_queryset(self): collection_uid = self.kwargs["collection_uid"] try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) except Collection.DoesNotExist: raise Http404("Collection does not exist") # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') @@ -312,7 +312,7 @@ class CollectionItemViewSet(BaseViewSet): @action_decorator(detail=True, methods=["GET"]) def revision(self, request, collection_uid=None, uid=None, *args, **kwargs): - col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) item = get_object_or_404(col.items, uid=uid) limit = int(request.GET.get("limit", 50)) @@ -386,7 +386,7 @@ class CollectionItemViewSet(BaseViewSet): with transaction.atomic(): # We need this for locking on the collection object collection_object = get_object_or_404( self.get_collection_queryset(Collection.objects).select_for_update(), # Lock writes on the collection - main_item__uid=collection_uid, + uid=collection_uid, ) if stoken is not None and stoken != collection_object.stoken: @@ -435,7 +435,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): return queryset.filter(members__user=user) def update(self, request, *args, collection_uid=None, collection_item_uid=None, uid=None, **kwargs): - col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) data = { @@ -459,7 +459,7 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): import os from django.views.static import serve - col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) chunk = get_object_or_404(col.chunks, uid=uid) @@ -487,7 +487,7 @@ class CollectionMemberViewSet(BaseViewSet): def get_queryset(self, queryset=None): collection_uid = self.kwargs["collection_uid"] try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) except Collection.DoesNotExist: raise Http404("Collection does not exist") @@ -525,7 +525,7 @@ class CollectionMemberViewSet(BaseViewSet): @action_decorator(detail=False, methods=["POST"], permission_classes=our_base_permission_classes) def leave(self, request, collection_uid=None, *args, **kwargs): collection_uid = self.kwargs["collection_uid"] - col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) + col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) member = col.members.get(user=request.user) self.perform_destroy(member) @@ -584,7 +584,7 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): collection_uid = serializer.validated_data.get("collection", {}).get("uid") try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) + collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) except Collection.DoesNotExist: raise Http404("Collection does not exist") From 0407320ad40d6419945751d3a08b3fcde0bc8d88 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 14 Dec 2020 13:48:05 +0200 Subject: [PATCH 337/511] Update changelog. --- ChangeLog.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index b47be77..e3c8232 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,8 @@ # Changelog +## Version 0.6.1 +* Collection: save the UID on the model to use the db for enforcing uniqueness + ## Version 0.6.0 * Fix stoken calculation performance - was VERY slow in some rare cases * Fix issues with host verification failing with a custom port - part 2 From 75712614791c86759d7b71ee2d75a49fb6802e5a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 14 Dec 2020 16:03:11 +0200 Subject: [PATCH 338/511] Remove unused imports. --- django_etebase/models.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index 11b6506..096371d 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -22,10 +22,7 @@ from django.db.models.functions import Coalesce, Greatest from django.utils.functional import cached_property from django.utils.crypto import get_random_string -from rest_framework import status - from . import app_settings -from .exceptions import EtebaseValidationError UidValidator = RegexValidator(regex=r"^[a-zA-Z0-9\-_]{20,}$", message="Not a valid UID") From 070abfcdd8b3ea1973ae38291eead680455622db Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 14 Dec 2020 16:12:34 +0200 Subject: [PATCH 339/511] Format using black. --- etebase_server/settings.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 9baf8d3..0e78c9b 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -96,10 +96,18 @@ WSGI_APPLICATION = "etebase_server.wsgi.application" # https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators AUTH_PASSWORD_VALIDATORS = [ - {"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",}, - {"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",}, - {"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",}, - {"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",}, + { + "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator", + }, + { + "NAME": "django.contrib.auth.password_validation.MinimumLengthValidator", + }, + { + "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator", + }, + { + "NAME": "django.contrib.auth.password_validation.NumericPasswordValidator", + }, ] From a19a982b1c09a790df8afa131f64a9e44d488fbe Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 14 Dec 2020 16:00:48 +0200 Subject: [PATCH 340/511] Sendfile: add a sendfile module based on django-sendfile2 --- django_etebase/sendfile/LICENSE | 28 ++++++ django_etebase/sendfile/README.md | 3 + django_etebase/sendfile/__init__.py | 1 + django_etebase/sendfile/backends/__init__.py | 0 .../sendfile/backends/development.py | 17 ++++ django_etebase/sendfile/backends/mod_wsgi.py | 17 ++++ django_etebase/sendfile/backends/nginx.py | 12 +++ django_etebase/sendfile/backends/simple.py | 60 +++++++++++++ django_etebase/sendfile/backends/xsendfile.py | 9 ++ django_etebase/sendfile/utils.py | 85 +++++++++++++++++++ 10 files changed, 232 insertions(+) create mode 100644 django_etebase/sendfile/LICENSE create mode 100644 django_etebase/sendfile/README.md create mode 100644 django_etebase/sendfile/__init__.py create mode 100644 django_etebase/sendfile/backends/__init__.py create mode 100644 django_etebase/sendfile/backends/development.py create mode 100644 django_etebase/sendfile/backends/mod_wsgi.py create mode 100644 django_etebase/sendfile/backends/nginx.py create mode 100644 django_etebase/sendfile/backends/simple.py create mode 100644 django_etebase/sendfile/backends/xsendfile.py create mode 100644 django_etebase/sendfile/utils.py diff --git a/django_etebase/sendfile/LICENSE b/django_etebase/sendfile/LICENSE new file mode 100644 index 0000000..4b733c8 --- /dev/null +++ b/django_etebase/sendfile/LICENSE @@ -0,0 +1,28 @@ +Copyright (c) 2011, Sensible Development. +Copyright (c) 2019, Matt Molyneaux +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + 3. Neither the name of Django Send File nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/django_etebase/sendfile/README.md b/django_etebase/sendfile/README.md new file mode 100644 index 0000000..aab5091 --- /dev/null +++ b/django_etebase/sendfile/README.md @@ -0,0 +1,3 @@ +Heavily inspired + code borrowed from: https://github.com/moggers87/django-sendfile2/ + +We just simplified and inlined it because we don't want another external dependency for distribution packagers to package, as well as need a much simpler version. diff --git a/django_etebase/sendfile/__init__.py b/django_etebase/sendfile/__init__.py new file mode 100644 index 0000000..4949aa5 --- /dev/null +++ b/django_etebase/sendfile/__init__.py @@ -0,0 +1 @@ +from .utils import sendfile # noqa diff --git a/django_etebase/sendfile/backends/__init__.py b/django_etebase/sendfile/backends/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/django_etebase/sendfile/backends/development.py b/django_etebase/sendfile/backends/development.py new file mode 100644 index 0000000..d321932 --- /dev/null +++ b/django_etebase/sendfile/backends/development.py @@ -0,0 +1,17 @@ +import os.path + +from django.views.static import serve + + +def sendfile(request, filename, **kwargs): + """ + Send file using Django dev static file server. + + .. warning:: + + Do not use in production. This is only to be used when developing and + is provided for convenience only + """ + dirname = os.path.dirname(filename) + basename = os.path.basename(filename) + return serve(request, basename, dirname) diff --git a/django_etebase/sendfile/backends/mod_wsgi.py b/django_etebase/sendfile/backends/mod_wsgi.py new file mode 100644 index 0000000..07ba3f1 --- /dev/null +++ b/django_etebase/sendfile/backends/mod_wsgi.py @@ -0,0 +1,17 @@ +from __future__ import absolute_import + +from django.http import HttpResponse + +from ..utils import _convert_file_to_url + + +def sendfile(request, filename, **kwargs): + response = HttpResponse() + response['Location'] = _convert_file_to_url(filename) + # need to destroy get_host() to stop django + # rewriting our location to include http, so that + # mod_wsgi is able to do the internal redirect + request.get_host = lambda: '' + request.build_absolute_uri = lambda location: location + + return response diff --git a/django_etebase/sendfile/backends/nginx.py b/django_etebase/sendfile/backends/nginx.py new file mode 100644 index 0000000..8764309 --- /dev/null +++ b/django_etebase/sendfile/backends/nginx.py @@ -0,0 +1,12 @@ +from __future__ import absolute_import + +from django.http import HttpResponse + +from ..utils import _convert_file_to_url + + +def sendfile(request, filename, **kwargs): + response = HttpResponse() + response['X-Accel-Redirect'] = _convert_file_to_url(filename) + + return response diff --git a/django_etebase/sendfile/backends/simple.py b/django_etebase/sendfile/backends/simple.py new file mode 100644 index 0000000..0549b20 --- /dev/null +++ b/django_etebase/sendfile/backends/simple.py @@ -0,0 +1,60 @@ +from email.utils import mktime_tz, parsedate_tz +import re + +from django.core.files.base import File +from django.http import HttpResponse, HttpResponseNotModified +from django.utils.http import http_date + + +def sendfile(request, filepath, **kwargs): + '''Use the SENDFILE_ROOT value composed with the path arrived as argument + to build an absolute path with which resolve and return the file contents. + + If the path points to a file out of the root directory (should cover both + situations with '..' and symlinks) then a 404 is raised. + ''' + statobj = filepath.stat() + + # Respect the If-Modified-Since header. + if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), + statobj.st_mtime, statobj.st_size): + return HttpResponseNotModified() + + with File(filepath.open('rb')) as f: + response = HttpResponse(f.chunks()) + + response["Last-Modified"] = http_date(statobj.st_mtime) + return response + + +def was_modified_since(header=None, mtime=0, size=0): + """ + Was something modified since the user last downloaded it? + + header + This is the value of the If-Modified-Since header. If this is None, + I'll just return True. + + mtime + This is the modification time of the item we're talking about. + + size + This is the size of the item we're talking about. + """ + try: + if header is None: + raise ValueError + matches = re.match(r"^([^;]+)(; length=([0-9]+))?$", header, + re.IGNORECASE) + header_date = parsedate_tz(matches.group(1)) + if header_date is None: + raise ValueError + header_mtime = mktime_tz(header_date) + header_len = matches.group(3) + if header_len and int(header_len) != size: + raise ValueError + if mtime > header_mtime: + raise ValueError + except (AttributeError, ValueError, OverflowError): + return True + return False diff --git a/django_etebase/sendfile/backends/xsendfile.py b/django_etebase/sendfile/backends/xsendfile.py new file mode 100644 index 0000000..74993ee --- /dev/null +++ b/django_etebase/sendfile/backends/xsendfile.py @@ -0,0 +1,9 @@ +from django.http import HttpResponse + + +def sendfile(request, filename, **kwargs): + filename = str(filename) + response = HttpResponse() + response['X-Sendfile'] = filename + + return response diff --git a/django_etebase/sendfile/utils.py b/django_etebase/sendfile/utils.py new file mode 100644 index 0000000..97c06d7 --- /dev/null +++ b/django_etebase/sendfile/utils.py @@ -0,0 +1,85 @@ +from functools import lru_cache +from importlib import import_module +from pathlib import Path, PurePath +from urllib.parse import quote +import logging + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.http import Http404 + +logger = logging.getLogger(__name__) + + +@lru_cache(maxsize=None) +def _get_sendfile(): + backend = getattr(settings, "SENDFILE_BACKEND", None) + if not backend: + raise ImproperlyConfigured("You must specify a value for SENDFILE_BACKEND") + module = import_module(backend) + return module.sendfile + + +def _convert_file_to_url(path): + try: + url_root = PurePath(getattr(settings, "SENDFILE_URL", None)) + except TypeError: + return path + + path_root = PurePath(settings.SENDFILE_ROOT) + path_obj = PurePath(path) + + relpath = path_obj.relative_to(path_root) + # Python 3.5: Path.resolve() has no `strict` kwarg, so use pathmod from an + # already instantiated Path object + url = relpath._flavour.pathmod.normpath(str(url_root / relpath)) + + return quote(str(url)) + + +def _sanitize_path(filepath): + try: + path_root = Path(getattr(settings, "SENDFILE_ROOT", None)) + except TypeError: + raise ImproperlyConfigured("You must specify a value for SENDFILE_ROOT") + + filepath_obj = Path(filepath) + + # get absolute path + # Python 3.5: Path.resolve() has no `strict` kwarg, so use pathmod from an + # already instantiated Path object + filepath_abs = Path(filepath_obj._flavour.pathmod.normpath(str(path_root / filepath_obj))) + + # if filepath_abs is not relative to path_root, relative_to throws an error + try: + filepath_abs.relative_to(path_root) + except ValueError: + raise Http404("{} wrt {} is impossible".format(filepath_abs, path_root)) + + return filepath_abs + + +def sendfile(request, filename, mimetype="application/octet-stream", encoding=None): + """ + Create a response to send file using backend configured in ``SENDFILE_BACKEND`` + + ``filename`` is the absolute path to the file to send. + """ + filepath_obj = _sanitize_path(filename) + logger.debug( + "filename '%s' requested \"\ + \"-> filepath '%s' obtained", + filename, + filepath_obj, + ) + _sendfile = _get_sendfile() + + if not filepath_obj.exists(): + raise Http404('"%s" does not exist' % filepath_obj) + + response = _sendfile(request, filepath_obj, mimetype=mimetype) + + response["Content-length"] = filepath_obj.stat().st_size + response["Content-Type"] = mimetype + + return response From 9559a0fd35bf0c95757a1ecadebf6418a6536515 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 14 Dec 2020 16:40:08 +0200 Subject: [PATCH 341/511] Chunk download: use the new sendfile to serve files. --- django_etebase/views.py | 11 ++--------- etebase_server/settings.py | 4 ++++ 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/django_etebase/views.py b/django_etebase/views.py index 44c3383..1de5ed7 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -36,6 +36,7 @@ import nacl.signing import nacl.secret import nacl.hash +from .sendfile import sendfile from .token_auth.models import AuthToken from .drf_msgpack.parsers import MessagePackParser @@ -456,19 +457,11 @@ class CollectionItemChunkViewSet(viewsets.ViewSet): @action_decorator(detail=True, methods=["GET"]) def download(self, request, collection_uid=None, collection_item_uid=None, uid=None, *args, **kwargs): - import os - from django.views.static import serve - col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) - # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) chunk = get_object_or_404(col.chunks, uid=uid) filename = chunk.chunkFile.path - dirname = os.path.dirname(filename) - basename = os.path.basename(filename) - - # FIXME: DO NOT USE! Use django-send file or etc instead. - return serve(request, basename, dirname) + return sendfile(request, filename) class CollectionMemberViewSet(BaseViewSet): diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 0e78c9b..325dca9 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -173,6 +173,10 @@ ETEBASE_API_AUTHENTICATORS = ( ) ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked" +# Efficient file streaming (for large files) +SENDFILE_BACKEND = "django_etebase.sendfile.backends.simple" +SENDFILE_ROOT = MEDIA_URL + # Make an `etebase_server_settings` module available to override settings. try: from etebase_server_settings import * From cd86c060b5242d82a3b2a62b0ad93bd5f42eb1c5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 15 Dec 2020 10:14:15 +0200 Subject: [PATCH 342/511] Collection: fix UID validation to return a Conflict error. --- django_etebase/serializers.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 9628986..223e48f 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -323,12 +323,17 @@ class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): if etag is not None: raise EtebaseValidationError("bad_etag", "etag is not null") + try: + instance.validate_unique() + except django_exceptions.ValidationError: + raise EtebaseValidationError( + "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT + ) instance.save() + main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) instance.main_item = main_item - - instance.full_clean() instance.save() process_revisions_for_item(main_item, revision_data) From 0585d6ee927ad5aa4a4baea3dac786db1a49b647 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 15 Dec 2020 13:58:59 +0200 Subject: [PATCH 343/511] Chunk file uploader: accept all kinds of media types. This restriction was unnecessary and annoying to deal with in clients. --- django_etebase/parsers.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/django_etebase/parsers.py b/django_etebase/parsers.py index c7fe58c..ed1e713 100644 --- a/django_etebase/parsers.py +++ b/django_etebase/parsers.py @@ -6,8 +6,6 @@ class ChunkUploadParser(FileUploadParser): Parser for chunk upload data. """ - media_type = "application/octet-stream" - def get_filename(self, stream, media_type, parser_context): """ Detects the uploaded file name. From 7ae172e44e71e7e0b541ba854dc4ac037523e949 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 16 Dec 2020 09:24:53 +0200 Subject: [PATCH 344/511] README: update contributors from github sponsors --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cfb0be9..3e0bd53 100644 --- a/README.md +++ b/README.md @@ -148,6 +148,13 @@ A quick summary can be found [on tldrlegal](https://tldrlegal.com/license/gnu-af For commercial licensing options, contact license@etebase.com -# Supporting Etebase +# Financially Supporting Etebase Please consider registering an account even if you self-host in order to support the development of Etebase, or visit the [contribution](https://www.etesync.com/contribute/) for more information on how to support the service. + +Become a financial contributor and help us sustain our community! + +## Contributors ($10 / month) + +[![ilovept](https://github.com/ilovept.png?size=40)](https://github.com/ilovept) +[![ryanleesipes](https://github.com/ryanleesipes.png?size=40)](https://github.com/ryanleesipes) From 3b4ba75930822d4cfd17cabc6b0d71b92c940150 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 16 Dec 2020 15:04:00 +0200 Subject: [PATCH 345/511] Chunk serialization: support not passing chunk content if exists. --- django_etebase/serializers.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 223e48f..eebae91 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -37,15 +37,14 @@ def process_revisions_for_item(item, revision_data): for chunk in chunks: uid = chunk[0] chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() - if len(chunk) > 1: - content = chunk[1] - # If the chunk already exists we assume it's fine. Otherwise, we upload it. - if chunk_obj is None: + content = chunk[1] if len(chunk) > 1 else None + # If the chunk already exists we assume it's fine. Otherwise, we upload it. + if chunk_obj is None: + if content is not None: chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) chunk_obj.save() - else: - if chunk_obj is None: + else: raise EtebaseValidationError("chunk_no_content", "Tried to create a new chunk without content") chunks_objs.append(chunk_obj) @@ -122,9 +121,10 @@ class ChunksField(serializers.RelatedField): return (obj.uid,) def to_internal_value(self, data): - if data[0] is None or data[1] is None: + content = data[1] if len(data) > 1 else None + if data[0] is None: raise EtebaseValidationError("no_null", "null is not allowed") - return (data[0], b64decode_or_bytes(data[1])) + return (data[0], b64decode_or_bytes(content) if content is not None else None) class BetterErrorsMixin: From 3fcea20d68f7a820386f1e6c95ce8528a760c4f4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 22 Dec 2020 12:46:42 +0200 Subject: [PATCH 346/511] Serializers: fully clean an object on signup. --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index eebae91..9408b78 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -518,7 +518,7 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): # Create the user and save the casing the user chose as the first name try: instance = create_user(**user_data, password=None, first_name=user_data["username"], view=view) - instance.clean_fields() + instance.full_clean() except EtebaseValidationError as e: raise e except django_exceptions.ValidationError as e: From 1cb84cfa6d8d10876ea4e5207963728dedcb77e9 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 22 Dec 2020 13:15:12 +0200 Subject: [PATCH 347/511] Serializers: cleanup how we handle validation errors. --- django_etebase/serializers.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 9408b78..24d78a4 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -149,10 +149,8 @@ class BetterErrorsMixin: ret.extend(self.flatten_errors("{}.{}".format(field_name, error_key), error)) else: for error in errors: - if hasattr(error, "detail"): - message = error.detail[0] - elif hasattr(error, "message"): - message = error.message + if error.messages: + message = error.messages[0] else: message = str(error) ret.append( From 7eb08d29462d6e663d5af57795fe1ed3f7b21652 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 22 Dec 2020 21:06:41 +0200 Subject: [PATCH 348/511] Collection invitation: fix the wrong field type --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 24d78a4..1e5c78e 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -386,7 +386,7 @@ class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSeriali style={"base_template": "input.html"}, ) collection = serializers.CharField(source="collection.uid") - fromUsername = BinaryBase64Field(source="fromMember.user.username", read_only=True) + fromUsername = serializers.CharField(source="fromMember.user.username", read_only=True) fromPubkey = BinaryBase64Field(source="fromMember.user.userinfo.pubkey", read_only=True) signedEncryptionKey = BinaryBase64Field() From 2de51b978abfac206f05624faacbbfa414cef32c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 24 Dec 2020 16:14:52 +0200 Subject: [PATCH 349/511] Serializers: fix an issue with handling validation errors. --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 1e5c78e..32a2757 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -149,7 +149,7 @@ class BetterErrorsMixin: ret.extend(self.flatten_errors("{}.{}".format(field_name, error_key), error)) else: for error in errors: - if error.messages: + if getattr(error, "messages", None): message = error.messages[0] else: message = str(error) From 5a6c8a1d05a885dd4a85b2ebde1b49690edbd0e9 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 10:53:01 +0200 Subject: [PATCH 350/511] Gracefully handle uploading the same revision This is needed so that immediately re-played requests don't fail. Consider for example the following scenario: a client makes a batch request that registers correctly on the server, but fails to return (e.g. a networking error after the request has been processed). The client would think that the request failed, but the server will already have the up to date information. This commit just returns a successful status if this request is sent again (by the client retrying the request) instead of returning a conflict. This however doesn't handle the case of a request failing, a modification being made by another client, and then the request being retried. This case will stay fail. --- django_etebase/serializers.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 32a2757..06bc8ad 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -239,6 +239,11 @@ class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): # We don't have to use select_for_update here because the unique constraint on current guards against # the race condition. But it's a good idea because it'll lock and wait rather than fail. current_revision = instance.revisions.filter(current=True).select_for_update().first() + + # If we are just re-uploading the same revision, consider it a succes and return. + if current_revision.uid == revision_data.get("uid"): + return instance + current_revision.current = None current_revision.save() From c2eb4fd30cd3dceca05514788a0c450045efc391 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 15:03:07 +0200 Subject: [PATCH 351/511] Pass generic context to callbacks instead of the whole view --- django_etebase/serializers.py | 13 +++++++++---- django_etebase/utils.py | 17 +++++++++++++---- django_etebase/views.py | 8 ++++---- 3 files changed, 26 insertions(+), 12 deletions(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index 06bc8ad..dce2fe6 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -20,7 +20,7 @@ from django.contrib.auth import get_user_model from django.db import IntegrityError, transaction from rest_framework import serializers, status from . import models -from .utils import get_user_queryset, create_user +from .utils import get_user_queryset, create_user, CallbackContext from .exceptions import EtebaseValidationError @@ -102,7 +102,7 @@ class CollectionTypeField(BinaryBase64Field): class UserSlugRelatedField(serializers.SlugRelatedField): def get_queryset(self): view = self.context.get("view", None) - return get_user_queryset(super().get_queryset(), view) + return get_user_queryset(super().get_queryset(), context=CallbackContext(view.kwargs)) def __init__(self, **kwargs): super().__init__(slug_field=User.USERNAME_FIELD, **kwargs) @@ -515,12 +515,17 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): with transaction.atomic(): try: view = self.context.get("view", None) - user_queryset = get_user_queryset(User.objects.all(), view) + user_queryset = get_user_queryset(User.objects.all(), context=CallbackContext(view.kwargs)) instance = user_queryset.get(**{User.USERNAME_FIELD: user_data["username"].lower()}) except User.DoesNotExist: # Create the user and save the casing the user chose as the first name try: - instance = create_user(**user_data, password=None, first_name=user_data["username"], view=view) + instance = create_user( + **user_data, + password=None, + first_name=user_data["username"], + context=CallbackContext(view.kwargs) + ) instance.full_clean() except EtebaseValidationError as e: raise e diff --git a/django_etebase/utils.py b/django_etebase/utils.py index e496a77..1c8654b 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -1,3 +1,6 @@ +import typing as t +from dataclasses import dataclass + from django.contrib.auth import get_user_model from django.core.exceptions import PermissionDenied @@ -7,18 +10,24 @@ from . import app_settings User = get_user_model() -def get_user_queryset(queryset, view): +@dataclass +class CallbackContext: + """Class for passing extra context to callbacks""" + + url_kwargs: t.Dict[str, t.Any] + + +def get_user_queryset(queryset, context: CallbackContext): custom_func = app_settings.GET_USER_QUERYSET_FUNC if custom_func is not None: - return custom_func(queryset, view) + return custom_func(queryset, context) return queryset -def create_user(*args, **kwargs): +def create_user(context: CallbackContext, *args, **kwargs): custom_func = app_settings.CREATE_USER_FUNC if custom_func is not None: return custom_func(*args, **kwargs) - _ = kwargs.pop("view") return User.objects.create_user(*args, **kwargs) diff --git a/django_etebase/views.py b/django_etebase/views.py index 1de5ed7..5a03aa4 100644 --- a/django_etebase/views.py +++ b/django_etebase/views.py @@ -73,7 +73,7 @@ from .serializers import ( UserInfoPubkeySerializer, UserSerializer, ) -from .utils import get_user_queryset +from .utils import get_user_queryset, CallbackContext from .exceptions import EtebaseValidationError from .parsers import ChunkUploadParser from .signals import user_signed_up @@ -598,7 +598,7 @@ class InvitationOutgoingViewSet(InvitationBaseViewSet): def fetch_user_profile(self, request, *args, **kwargs): username = request.GET.get("username") kwargs = {User.USERNAME_FIELD: username.lower()} - user = get_object_or_404(get_user_queryset(User.objects.all(), self), **kwargs) + user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(self.kwargs)), **kwargs) user_info = get_object_or_404(UserInfo.objects.all(), owner=user) serializer = UserInfoPubkeySerializer(user_info) return Response(serializer.data) @@ -642,7 +642,7 @@ class AuthenticationViewSet(viewsets.ViewSet): ) def get_queryset(self): - return get_user_queryset(User.objects.all(), self) + return get_user_queryset(User.objects.all(), CallbackContext(self.kwargs)) def get_serializer_context(self): return {"request": self.request, "format": self.format_kwarg, "view": self} @@ -837,7 +837,7 @@ class TestAuthenticationViewSet(viewsets.ViewSet): return HttpResponseBadRequest("Only allowed in debug mode.") with transaction.atomic(): - user_queryset = get_user_queryset(User.objects.all(), self) + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(self.kwargs)) user = get_object_or_404(user_queryset, username=request.data.get("user").get("username")) # Only allow test users for extra safety From 70619fc1c7b13f0311170d6087b5b42ae876cb49 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 15:14:42 +0200 Subject: [PATCH 352/511] Fix unbound variable warning. --- django_etebase/serializers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py index dce2fe6..26ac5a7 100644 --- a/django_etebase/serializers.py +++ b/django_etebase/serializers.py @@ -513,8 +513,8 @@ class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): user_data = validated_data.pop("user") with transaction.atomic(): + view = self.context.get("view", None) try: - view = self.context.get("view", None) user_queryset = get_user_queryset(User.objects.all(), context=CallbackContext(view.kwargs)) instance = user_queryset.get(**{User.USERNAME_FIELD: user_data["username"].lower()}) except User.DoesNotExist: From c1534f6587dfc25569fb270b8b7b43984da96f3e Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Wed, 23 Dec 2020 16:29:08 -0500 Subject: [PATCH 353/511] first commit --- etebase_fastapi/__init__.py | 0 etebase_fastapi/app.py | 29 ++++ etebase_fastapi/authentication.py | 251 ++++++++++++++++++++++++++++++ etebase_fastapi/collections.py | 0 etebase_fastapi/execptions.py | 42 +++++ etebase_fastapi/msgpack.py | 63 ++++++++ requirements.in/base.txt | 2 + requirements.txt | 30 ++-- 8 files changed, 405 insertions(+), 12 deletions(-) create mode 100644 etebase_fastapi/__init__.py create mode 100644 etebase_fastapi/app.py create mode 100644 etebase_fastapi/authentication.py create mode 100644 etebase_fastapi/collections.py create mode 100644 etebase_fastapi/execptions.py create mode 100644 etebase_fastapi/msgpack.py diff --git a/etebase_fastapi/__init__.py b/etebase_fastapi/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py new file mode 100644 index 0000000..0ee7aae --- /dev/null +++ b/etebase_fastapi/app.py @@ -0,0 +1,29 @@ +import os + +from django.core.wsgi import get_wsgi_application +from fastapi.middleware.cors import CORSMiddleware + +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") +application = get_wsgi_application() +from fastapi import FastAPI, Request + +from .execptions import CustomHttpException +from .authentication import authentication_router +from .msgpack import MsgpackResponse + +app = FastAPI() +app.include_router(authentication_router, prefix="/api/v1/authentication") +app.add_middleware( + CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] +) + + +@app.exception_handler(CustomHttpException) +async def custom_exception_handler(request: Request, exc: CustomHttpException): + return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) + + +if __name__ == "__main__": + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=8080) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py new file mode 100644 index 0000000..b1a3272 --- /dev/null +++ b/etebase_fastapi/authentication.py @@ -0,0 +1,251 @@ +import dataclasses +import typing as t +from datetime import datetime +from functools import cached_property + +import nacl +import nacl.encoding +import nacl.hash +import nacl.secret +import nacl.signing +from asgiref.sync import sync_to_async +from django.conf import settings +from django.contrib.auth import get_user_model, user_logged_out, user_logged_in +from django.utils import timezone +from fastapi import APIRouter, Depends, status, Request, Response +from fastapi.security import APIKeyHeader +from pydantic import BaseModel + +from django_etebase import app_settings +from django_etebase.models import UserInfo +from django_etebase.serializers import UserSerializer +from django_etebase.token_auth.models import AuthToken +from django_etebase.token_auth.models import get_default_expiry +from django_etebase.views import msgpack_encode, msgpack_decode +from .execptions import AuthenticationFailed +from .msgpack import MsgpackResponse, MsgpackRoute + +User = get_user_model() +token_scheme = APIKeyHeader(name="Authorization") +AUTO_REFRESH = True +MIN_REFRESH_INTERVAL = 60 +authentication_router = APIRouter(route_class=MsgpackRoute) + + +@dataclasses.dataclass(frozen=True) +class AuthData: + user: User + token: AuthToken + + +class LoginChallengeData(BaseModel): + username: str + + +class LoginResponse(BaseModel): + username: str + challenge: bytes + host: str + action: t.Literal["login", "changePassword"] + + +class Authentication(BaseModel): + response: bytes + signature: bytes + + +class Login(Authentication): + @cached_property + def response_data(self) -> LoginResponse: + return LoginResponse(**msgpack_decode(self.response)) + + +class ChangePasswordResponse(LoginResponse): + loginPubkey: bytes + encryptedContent: bytes + + +class ChangePassword(Authentication): + @cached_property + def response_data(self) -> ChangePasswordResponse: + return ChangePasswordResponse(**msgpack_decode(self.response)) + + +def __renew_token(auth_token: AuthToken): + current_expiry = auth_token.expiry + new_expiry = get_default_expiry() + # Throttle refreshing of token to avoid db writes + delta = (new_expiry - current_expiry).total_seconds() + if delta > MIN_REFRESH_INTERVAL: + auth_token.expiry = new_expiry + auth_token.save(update_fields=("expiry",)) + + +@sync_to_async +def __get_authenticated_user(api_token: str): + api_token = api_token.split()[1] + try: + token: AuthToken = AuthToken.objects.select_related("user").get(key=api_token) + except AuthToken.DoesNotExist: + raise AuthenticationFailed(detail="Invalid token.") + if not token.user.is_active: + raise AuthenticationFailed(detail="User inactive or deleted.") + + if token.expiry is not None: + if token.expiry < timezone.now(): + token.delete() + raise AuthenticationFailed(detail="Invalid token.") + + if AUTO_REFRESH: + __renew_token(token) + + return token.user, token + + +async def get_auth_data(api_token: str = Depends(token_scheme)) -> AuthData: + user, token = await __get_authenticated_user(api_token) + return AuthData(user, token) + + +async def get_authenticated_user(api_token: str = Depends(token_scheme)) -> User: + user, token = await __get_authenticated_user(api_token) + return user + + +@sync_to_async +def __get_login_user(username: str) -> User: + kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} + try: + user = User.objects.get(**kwargs) + if not hasattr(user, "userinfo"): + raise AuthenticationFailed(code="user_not_init", detail="User not properly init") + return user + except User.DoesNotExist: + raise AuthenticationFailed(code="user_not_found", detail="User not found") + + +async def get_login_user(challenge: LoginChallengeData) -> User: + user = await __get_login_user(challenge.username) + return user + + +def get_encryption_key(salt): + key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) + return nacl.hash.blake2b( + b"", + key=key, + salt=salt[: nacl.hash.BLAKE2B_SALTBYTES], + person=b"etebase-auth", + encoder=nacl.encoding.RawEncoder, + ) + + +@sync_to_async +def save_changed_password(data: ChangePassword, user: User): + response_data = data.response_data + user_info: UserInfo = user.userinfo + user_info.loginPubkey = response_data.loginPubkey + user_info.encryptedContent = response_data.encryptedContent + user_info.save() + + +@sync_to_async +def login_response_data(user: User): + return { + "token": AuthToken.objects.create(user=user).key, + "user": UserSerializer(user).data, + } + + +@sync_to_async +def send_user_logged_in_async(user: User, request: Request): + user_logged_in.send(sender=user.__class__, request=request, user=user) + + +@sync_to_async +def send_user_logged_out_async(user: User, request: Request): + user_logged_out.send(sender=user.__class__, request=request, user=user) + + +@sync_to_async +def validate_login_request( + validated_data: LoginResponse, + challenge_sent_to_user: Authentication, + user: User, + expected_action: str, + host_from_request: str, +) -> t.Optional[MsgpackResponse]: + + enc_key = get_encryption_key(bytes(user.userinfo.salt)) + box = nacl.secret.SecretBox(enc_key) + challenge_data = msgpack_decode(box.decrypt(validated_data.challenge)) + now = int(datetime.now().timestamp()) + if validated_data.action != expected_action: + content = { + "code": "wrong_action", + "detail": 'Expected "{}" but got something else'.format(challenge_sent_to_user.response), + } + return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: + content = {"code": "challenge_expired", "detail": "Login challenge has expired"} + return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + elif challenge_data["userId"] != user.id: + content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} + return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request: + detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(validated_data.host, host_from_request) + content = {"code": "wrong_host", "detail": detail} + return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + + verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) + + try: + verify_key.verify(challenge_sent_to_user.response, challenge_sent_to_user.signature) + except nacl.exceptions.BadSignatureError: + return MsgpackResponse( + {"code": "login_bad_signature", "detail": "Wrong password for user."}, + status_code=status.HTTP_401_UNAUTHORIZED, + ) + + return None + + +@authentication_router.post("/login_challenge/") +async def login_challenge(user: User = Depends(get_login_user)): + enc_key = get_encryption_key(user.userinfo.salt) + box = nacl.secret.SecretBox(enc_key) + challenge_data = { + "timestamp": int(datetime.now().timestamp()), + "userId": user.id, + } + challenge = bytes(box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder)) + return MsgpackResponse({"salt": user.userinfo.salt, "version": user.userinfo.version, "challenge": challenge}) + + +@authentication_router.post("/login/") +async def login(data: Login, request: Request): + user = await get_login_user(LoginChallengeData(username=data.response_data.username)) + host = request.headers.get("Host") + bad_login_response = await validate_login_request(data.response_data, data, user, "login", host) + if bad_login_response is not None: + return bad_login_response + data = await login_response_data(user) + await send_user_logged_in_async(user, request) + return MsgpackResponse(data, status_code=status.HTTP_200_OK) + + +@authentication_router.post("/logout/") +async def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): + await sync_to_async(auth_data.token.delete)() + await send_user_logged_out_async(auth_data.user, request) + return Response(status_code=status.HTTP_204_NO_CONTENT) + + +@authentication_router.post("/change_password/") +async def change_password(data: ChangePassword, request: Request, user: User = Depends(get_authenticated_user)): + host = request.headers.get("Host") + bad_login_response = await validate_login_request(data.response_data, data, user, "changePassword", host) + if bad_login_response is not None: + return bad_login_response + await save_changed_password(data, user) + return Response(status_code=status.HTTP_204_NO_CONTENT) diff --git a/etebase_fastapi/collections.py b/etebase_fastapi/collections.py new file mode 100644 index 0000000..e69de29 diff --git a/etebase_fastapi/execptions.py b/etebase_fastapi/execptions.py new file mode 100644 index 0000000..8808f5d --- /dev/null +++ b/etebase_fastapi/execptions.py @@ -0,0 +1,42 @@ +from fastapi import status + + +class CustomHttpException(Exception): + def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): + self.status_code = status_code + self.code = code + self.detail = detail + + @property + def as_dict(self) -> dict: + return {"code": self.code, "detail": self.detail} + + +class AuthenticationFailed(CustomHttpException): + def __init__( + self, + code="authentication_failed", + detail: str = "Incorrect authentication credentials.", + status_code: int = status.HTTP_401_UNAUTHORIZED, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + +class NotAuthenticated(CustomHttpException): + def __init__( + self, + code="not_authenticated", + detail: str = "Authentication credentials were not provided.", + status_code: int = status.HTTP_401_UNAUTHORIZED, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + +class PermissionDenied(CustomHttpException): + def __init__( + self, + code="permission_denied", + detail: str = "You do not have permission to perform this action.", + status_code: int = status.HTTP_403_FORBIDDEN, + ): + super().__init__(code=code, detail=detail, status_code=status_code) diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py new file mode 100644 index 0000000..53e18cb --- /dev/null +++ b/etebase_fastapi/msgpack.py @@ -0,0 +1,63 @@ +import typing as t +import msgpack +from fastapi.routing import APIRoute, get_request_handler +from starlette.requests import Request +from starlette.responses import Response + + +class MsgpackRequest(Request): + media_type = "application/msgpack" + + async def json(self) -> bytes: + if not hasattr(self, "_json"): + body = await super().body() + self._json = msgpack.unpackb(body, raw=False) + return self._json + + +class MsgpackResponse(Response): + media_type = "application/msgpack" + + def render(self, content: t.Any) -> bytes: + return msgpack.packb(content, use_bin_type=True) + + +class MsgpackRoute(APIRoute): + # keep track of content-type -> request classes + REQUESTS_CLASSES = {MsgpackRequest.media_type: MsgpackRequest} + # keep track of content-type -> response classes + ROUTES_HANDLERS_CLASSES = {MsgpackResponse.media_type: MsgpackResponse} + + def _get_media_type_route_handler(self, media_type): + return get_request_handler( + dependant=self.dependant, + body_field=self.body_field, + status_code=self.status_code, + # use custom response class or fallback on default self.response_class + response_class=self.ROUTES_HANDLERS_CLASSES.get(media_type, self.response_class), + response_field=self.secure_cloned_response_field, + response_model_include=self.response_model_include, + response_model_exclude=self.response_model_exclude, + response_model_by_alias=self.response_model_by_alias, + response_model_exclude_unset=self.response_model_exclude_unset, + response_model_exclude_defaults=self.response_model_exclude_defaults, + response_model_exclude_none=self.response_model_exclude_none, + dependency_overrides_provider=self.dependency_overrides_provider, + ) + + def get_route_handler(self) -> t.Callable: + async def custom_route_handler(request: Request) -> Response: + + content_type = request.headers.get("Content-Type") + try: + request_cls = self.REQUESTS_CLASSES[content_type] + request = request_cls(request.scope, request.receive) + except KeyError: + # nothing registered to handle content_type, process given requests as-is + pass + + accept = request.headers.get("Accept") + route_handler = self._get_media_type_route_handler(accept) + return await route_handler(request) + + return custom_route_handler diff --git a/requirements.in/base.txt b/requirements.in/base.txt index 7d5bf7e..ca8dd94 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -5,3 +5,5 @@ drf-nested-routers msgpack psycopg2-binary pynacl +fastapi +uvicorn \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index f6c8ed4..3d19eaf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,16 +4,22 @@ # # pip-compile --output-file=requirements.txt requirements.in/base.txt # -asgiref==3.2.10 # via django -cffi==1.14.0 # via pynacl -django-cors-headers==3.2.1 # via -r requirements.in/base.txt -django==3.1.1 # via -r requirements.in/base.txt, django-cors-headers, djangorestframework, drf-nested-routers -djangorestframework==3.11.0 # via -r requirements.in/base.txt, drf-nested-routers -drf-nested-routers==0.91 # via -r requirements.in/base.txt -msgpack==1.0.0 # via -r requirements.in/base.txt -psycopg2-binary==2.8.4 # via -r requirements.in/base.txt +asgiref==3.3.1 # via django +cffi==1.14.4 # via pynacl +click==7.1.2 # via uvicorn +django-cors-headers==3.6.0 # via -r requirements.in/base.txt +django==3.1.4 # via -r requirements.in/base.txt, django-cors-headers, djangorestframework, drf-nested-routers +djangorestframework==3.12.2 # via -r requirements.in/base.txt, drf-nested-routers +drf-nested-routers==0.92.5 # via -r requirements.in/base.txt +fastapi==0.63.0 # via -r requirements.in/base.txt +h11==0.11.0 # via uvicorn +msgpack==1.0.2 # via -r requirements.in/base.txt +psycopg2-binary==2.8.6 # via -r requirements.in/base.txt pycparser==2.20 # via cffi -pynacl==1.3.0 # via -r requirements.in/base.txt -pytz==2019.3 # via django -six==1.14.0 # via pynacl -sqlparse==0.3.0 # via django +pydantic==1.7.3 # via fastapi +pynacl==1.4.0 # via -r requirements.in/base.txt +pytz==2020.4 # via django +six==1.15.0 # via pynacl +sqlparse==0.4.1 # via django +starlette==0.13.6 # via fastapi +uvicorn==0.13.2 # via -r requirements.in/base.txt From 25cb4fec0cbe16d062123460d3ca2ed2669b3738 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 11:10:43 +0200 Subject: [PATCH 354/511] msgpack.py: allow pydantic BaseModel in content --- etebase_fastapi/msgpack.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index 53e18cb..399f3d0 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -1,6 +1,7 @@ import typing as t import msgpack from fastapi.routing import APIRoute, get_request_handler +from pydantic import BaseModel from starlette.requests import Request from starlette.responses import Response @@ -19,6 +20,8 @@ class MsgpackResponse(Response): media_type = "application/msgpack" def render(self, content: t.Any) -> bytes: + if isinstance(content, BaseModel): + content = content.dict() return msgpack.packb(content, use_bin_type=True) From 16a99f02ea9465fb5d5690820b38c4055cbc66ea Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 11:12:22 +0200 Subject: [PATCH 355/511] snapshot --- etebase_fastapi/app.py | 8 +++- etebase_fastapi/authentication.py | 3 ++ etebase_fastapi/collection.py | 72 +++++++++++++++++++++++++++++++ etebase_fastapi/collections.py | 0 4 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 etebase_fastapi/collection.py delete mode 100644 etebase_fastapi/collections.py diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 0ee7aae..acfb42f 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -9,10 +9,14 @@ from fastapi import FastAPI, Request from .execptions import CustomHttpException from .authentication import authentication_router +from .collection import collection_router from .msgpack import MsgpackResponse app = FastAPI() -app.include_router(authentication_router, prefix="/api/v1/authentication") +VERSION = "v1" +BASE_PATH = f"/api/{VERSION}" +app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication") +app.include_router(collection_router, prefix=f"{BASE_PATH}/collection") app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] ) @@ -26,4 +30,4 @@ async def custom_exception_handler(request: Request, exc: CustomHttpException): if __name__ == "__main__": import uvicorn - uvicorn.run(app, host="0.0.0.0", port=8080) + uvicorn.run(app, host="0.0.0.0", port=8000) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index b1a3272..697f3f4 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -50,6 +50,9 @@ class LoginResponse(BaseModel): class Authentication(BaseModel): + class Config: + keep_untouched = (cached_property,) + response: bytes signature: bytes diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py new file mode 100644 index 0000000..ec0125d --- /dev/null +++ b/etebase_fastapi/collection.py @@ -0,0 +1,72 @@ +import typing as t + +from django.contrib.auth import get_user_model +from django.db.models import Q +from django.db.models import QuerySet +from fastapi import APIRouter, Depends +from pydantic import BaseModel +from asgiref.sync import sync_to_async + +from django_etebase.models import Collection, Stoken, AccessLevels, CollectionMember +from .authentication import get_authenticated_user +from .msgpack import MsgpackRoute, MsgpackResponse + +User = get_user_model() +collection_router = APIRouter(route_class=MsgpackRoute) +default_queryset = Collection.objects.all() + + +class ListMulti(BaseModel): + collectionTypes: t.List[bytes] + + +class CollectionItemOut(BaseModel): + uid: str + + +class CollectionOut(BaseModel): + collectionKey: bytes + collectionType: bytes + accessLevel: AccessLevels + stoken: str + item: CollectionItemOut + + @classmethod + def from_orm_user(cls: t.Type["CollectionOut"], obj: Collection, user: User) -> "CollectionOut": + member: CollectionMember = obj.members.get(user=user) + collection_type = member.collectionType + return cls( + collectionType=collection_type and collection_type.uid, + collectionKey=member.encryptionKey, + accessLevel=member.accessLevel, + stoken=obj.stoken, + item=CollectionItemOut(uid=obj.main_item.uid), + ) + + +class ListResponse(BaseModel): + data: t.List[CollectionOut] + stoken: t.Optional[str] + done: bool + + +@sync_to_async +def list_common(queryset: QuerySet, stoken: t.Optional[str], user: User) -> MsgpackResponse: + data: t.List[CollectionOut] = [CollectionOut.from_orm_user(item, user) for item in queryset] + ret = ListResponse(data=data, stoken=stoken, done=True) + return MsgpackResponse(content=ret) + + +def get_collection_queryset(user: User, queryset: QuerySet) -> QuerySet: + return queryset.filter(members__user=user) + + +@collection_router.post("/list_multi/") +async def list_multi(limit: int, data: ListMulti, user: User = Depends(get_authenticated_user)): + queryset = get_collection_queryset(user, default_queryset) + # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") + queryset = queryset.filter( + Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) + ) + response = await list_common(queryset, None, user) + return response diff --git a/etebase_fastapi/collections.py b/etebase_fastapi/collections.py deleted file mode 100644 index e69de29..0000000 From f70e2d80a64d33ad485dbb58f0cfa113cdc72f4b Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 11:52:43 +0200 Subject: [PATCH 356/511] stoken_handler.py --- etebase_fastapi/stoken_handler.py | 61 +++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 etebase_fastapi/stoken_handler.py diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py new file mode 100644 index 0000000..c840f0e --- /dev/null +++ b/etebase_fastapi/stoken_handler.py @@ -0,0 +1,61 @@ +import typing as t + +from django.db.models import QuerySet +from fastapi import status + +from django_etebase.exceptions import EtebaseValidationError +from django_etebase.models import Stoken + +# TODO missing stoken_annotation type +StokenAnnotation = t.Any + + +def get_stoken_obj(stoken: t.Optional[str]): + if stoken is not None: + try: + return Stoken.objects.get(uid=stoken) + except Stoken.DoesNotExist: + raise EtebaseValidationError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) + + return None + + +def filter_by_stoken( + stoken: t.Optional[str], queryset: QuerySet, stoken_annotation: StokenAnnotation +) -> t.Tuple[QuerySet, t.Optional[str]]: + stoken_rev = get_stoken_obj(stoken) + + queryset = queryset.annotate(max_stoken=stoken_annotation).order_by("max_stoken") + + if stoken_rev is not None: + queryset = queryset.filter(max_stoken__gt=stoken_rev.id) + + return queryset, stoken_rev + + +def get_queryset_stoken(queryset: list) -> t.Optional[Stoken]: + maxid = -1 + for row in queryset: + rowmaxid = getattr(row, "max_stoken") or -1 + maxid = max(maxid, rowmaxid) + new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid) + + return new_stoken or None + + +def filter_by_stoken_and_limit( + stoken: t.Optional[str], limit: int, queryset: QuerySet, stoken_annotation: StokenAnnotation +) -> t.Tuple[list, t.Optional[Stoken], bool]: + + queryset, stoken_rev = filter_by_stoken(stoken=stoken, queryset=queryset, stoken_annotation=stoken_annotation) + + result = list(queryset[: limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + new_stoken_obj = get_queryset_stoken(result) or stoken_rev + + return result, new_stoken_obj, done From 7d864594802b5f7f0d8e23ab383f37e68a79df79 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 11:53:11 +0200 Subject: [PATCH 357/511] collection.pyL list_multi --- etebase_fastapi/collection.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index ec0125d..7c9aca8 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -10,6 +10,7 @@ from asgiref.sync import sync_to_async from django_etebase.models import Collection, Stoken, AccessLevels, CollectionMember from .authentication import get_authenticated_user from .msgpack import MsgpackRoute, MsgpackResponse +from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_queryset_stoken User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -51,9 +52,11 @@ class ListResponse(BaseModel): @sync_to_async -def list_common(queryset: QuerySet, stoken: t.Optional[str], user: User) -> MsgpackResponse: +def list_common(queryset: QuerySet, user: User, stoken: t.Optional[str], limit: int) -> MsgpackResponse: + result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, Collection.stoken_annotation) + new_stoken = new_stoken_obj and new_stoken_obj.uid data: t.List[CollectionOut] = [CollectionOut.from_orm_user(item, user) for item in queryset] - ret = ListResponse(data=data, stoken=stoken, done=True) + ret = ListResponse(data=data, stoken=new_stoken, done=done) return MsgpackResponse(content=ret) @@ -62,11 +65,13 @@ def get_collection_queryset(user: User, queryset: QuerySet) -> QuerySet: @collection_router.post("/list_multi/") -async def list_multi(limit: int, data: ListMulti, user: User = Depends(get_authenticated_user)): +async def list_multi( + data: ListMulti, stoken: t.Optional[str] = None, limit: int = 50, user: User = Depends(get_authenticated_user) +): queryset = get_collection_queryset(user, default_queryset) # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) ) - response = await list_common(queryset, None, user) + response = await list_common(queryset, user, stoken, limit) return response From c90e92b0f00aea03db62edd803925cfb8ceb4ea5 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 12:36:06 +0200 Subject: [PATCH 358/511] snapshot --- etebase_fastapi/authentication.py | 52 +++++++++++++++++++++++++++++++ etebase_fastapi/execptions.py | 46 +++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 697f3f4..3ae6c4b 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -2,6 +2,7 @@ import dataclasses import typing as t from datetime import datetime from functools import cached_property +from django.core import exceptions as django_exceptions import nacl import nacl.encoding @@ -11,16 +12,19 @@ import nacl.signing from asgiref.sync import sync_to_async from django.conf import settings from django.contrib.auth import get_user_model, user_logged_out, user_logged_in +from django.db import transaction from django.utils import timezone from fastapi import APIRouter, Depends, status, Request, Response from fastapi.security import APIKeyHeader from pydantic import BaseModel from django_etebase import app_settings +from django_etebase.exceptions import EtebaseValidationError from django_etebase.models import UserInfo from django_etebase.serializers import UserSerializer from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry +from django_etebase.utils import create_user from django_etebase.views import msgpack_encode, msgpack_decode from .execptions import AuthenticationFailed from .msgpack import MsgpackResponse, MsgpackRoute @@ -74,6 +78,19 @@ class ChangePassword(Authentication): return ChangePasswordResponse(**msgpack_decode(self.response)) +class UserSignup(BaseModel): + username: str + email: str + + +class SignupIn(BaseModel): + user: UserSignup + salt: bytes + loginPubkey: bytes + pubkey: bytes + encryptedContent: bytes + + def __renew_token(auth_token: AuthToken): current_expiry = auth_token.expiry new_expiry = get_default_expiry() @@ -252,3 +269,38 @@ async def change_password(data: ChangePassword, request: Request, user: User = D return bad_login_response await save_changed_password(data, user) return Response(status_code=status.HTTP_204_NO_CONTENT) + + +@sync_to_async +def signup_save(data: SignupIn): + user_data = data.user + with transaction.atomic(): + try: + # XXX-TOM + # view = self.context.get("view", None) + # user_queryset = get_user_queryset(User.objects.all(), view) + user_queryset = User.objects.all() + instance = user_queryset.get(**{User.USERNAME_FIELD: user_data.username.lower()}) + except User.DoesNotExist: + # Create the user and save the casing the user chose as the first name + try: + # XXX-TOM + instance = create_user(**user_data.dict(), password=None, first_name=user_data.username, view=None) + instance.full_clean() + except EtebaseValidationError as e: + raise e + except django_exceptions.ValidationError as e: + self.transform_validation_error("user", e) + except Exception as e: + raise EtebaseValidationError("generic", str(e)) + + if hasattr(instance, "userinfo"): + raise EtebaseValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) + + models.UserInfo.objects.create(**validated_data, owner=instance) + return instance + + +@authentication_router.post("/signup/") +async def signup(data: SignupIn): + pass diff --git a/etebase_fastapi/execptions.py b/etebase_fastapi/execptions.py index 8808f5d..2b35634 100644 --- a/etebase_fastapi/execptions.py +++ b/etebase_fastapi/execptions.py @@ -1,5 +1,7 @@ from fastapi import status +from django_etebase.exceptions import EtebaseValidationError + class CustomHttpException(Exception): def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): @@ -40,3 +42,47 @@ class PermissionDenied(CustomHttpException): status_code: int = status.HTTP_403_FORBIDDEN, ): super().__init__(code=code, detail=detail, status_code=status_code) + + +class ValidationError(CustomHttpException): + def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): + super().__init__(code=code, detail=detail, status_code=status_code) + + +def flatten_errors(field_name, errors): + ret = [] + if isinstance(errors, dict): + for error_key in errors: + error = errors[error_key] + ret.extend(flatten_errors("{}.{}".format(field_name, error_key), error)) + else: + for error in errors: + if error.messages: + message = error.messages[0] + else: + message = str(error) + ret.append( + { + "field": field_name, + "code": error.code, + "detail": message, + } + ) + return ret + + +def transform_validation_error(prefix, err): + if hasattr(err, "error_dict"): + errors = flatten_errors(prefix, err.error_dict) + elif not hasattr(err, "message"): + errors = flatten_errors(prefix, err.error_list) + else: + raise EtebaseValidationError(err.code, err.message) + raise ValidationError(code="field_errors", detail="Field validations failed.") + raise serializers.ValidationError( + { + "code": "field_errors", + "detail": "Field validations failed.", + "errors": errors, + } + ) From 72d4a725f5c969bff271b34239363a7c3ee909e3 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 13:21:20 +0200 Subject: [PATCH 359/511] validation errors --- etebase_fastapi/authentication.py | 24 ++++++++++----- etebase_fastapi/execptions.py | 51 ++++++++++++++++++++----------- 2 files changed, 51 insertions(+), 24 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 3ae6c4b..287b46e 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -18,15 +18,16 @@ from fastapi import APIRouter, Depends, status, Request, Response from fastapi.security import APIKeyHeader from pydantic import BaseModel -from django_etebase import app_settings +from django_etebase import app_settings, models from django_etebase.exceptions import EtebaseValidationError from django_etebase.models import UserInfo from django_etebase.serializers import UserSerializer +from django_etebase.signals import user_signed_up from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user from django_etebase.views import msgpack_encode, msgpack_decode -from .execptions import AuthenticationFailed +from .execptions import AuthenticationFailed, transform_validation_error, ValidationError from .msgpack import MsgpackResponse, MsgpackRoute User = get_user_model() @@ -272,7 +273,7 @@ async def change_password(data: ChangePassword, request: Request, user: User = D @sync_to_async -def signup_save(data: SignupIn): +def signup_save(data: SignupIn) -> User: user_data = data.user with transaction.atomic(): try: @@ -290,17 +291,26 @@ def signup_save(data: SignupIn): except EtebaseValidationError as e: raise e except django_exceptions.ValidationError as e: - self.transform_validation_error("user", e) + transform_validation_error("user", e) except Exception as e: raise EtebaseValidationError("generic", str(e)) if hasattr(instance, "userinfo"): - raise EtebaseValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) + raise ValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) - models.UserInfo.objects.create(**validated_data, owner=instance) + models.UserInfo.objects.create(**data.dict(exclude={"user"}), owner=instance) return instance +@sync_to_async +def send_user_signed_up_async(user: User, request): + user_signed_up.send(sender=user.__class__, request=request, user=user) + + @authentication_router.post("/signup/") async def signup(data: SignupIn): - pass + user = await signup_save(data) + # XXX-TOM + data = await login_response_data(user) + await send_user_signed_up_async(user, None) + return MsgpackResponse(content=data, status_code=status.HTTP_201_CREATED) diff --git a/etebase_fastapi/execptions.py b/etebase_fastapi/execptions.py index 2b35634..fa76c45 100644 --- a/etebase_fastapi/execptions.py +++ b/etebase_fastapi/execptions.py @@ -1,8 +1,23 @@ from fastapi import status +import typing as t + +from pydantic import BaseModel from django_etebase.exceptions import EtebaseValidationError +class ValidationErrorField(BaseModel): + field: str + code: str + detail: str + + +class ValidationErrorOut(BaseModel): + code: str + detail: str + errors: t.Optional[t.List[ValidationErrorField]] + + class CustomHttpException(Exception): def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): self.status_code = status_code @@ -44,12 +59,27 @@ class PermissionDenied(CustomHttpException): super().__init__(code=code, detail=detail, status_code=status_code) +from django_etebase.exceptions import EtebaseValidationError + + class ValidationError(CustomHttpException): - def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): + def __init__( + self, + code: str, + detail: str, + status_code: int = status.HTTP_400_BAD_REQUEST, + field: t.Optional[str] = None, + errors: t.Optional[t.List["ValidationError"]] = None, + ): + self.errors = errors super().__init__(code=code, detail=detail, status_code=status_code) + @property + def as_dict(self) -> dict: + return ValidationErrorOut(code=self.code, errors=self.errors, detail=self.detail).dict() + -def flatten_errors(field_name, errors): +def flatten_errors(field_name, errors) -> t.List[ValidationError]: ret = [] if isinstance(errors, dict): for error_key in errors: @@ -61,13 +91,7 @@ def flatten_errors(field_name, errors): message = error.messages[0] else: message = str(error) - ret.append( - { - "field": field_name, - "code": error.code, - "detail": message, - } - ) + ret.append(dict(code=error.code, detail=message, field=field_name)) return ret @@ -78,11 +102,4 @@ def transform_validation_error(prefix, err): errors = flatten_errors(prefix, err.error_list) else: raise EtebaseValidationError(err.code, err.message) - raise ValidationError(code="field_errors", detail="Field validations failed.") - raise serializers.ValidationError( - { - "code": "field_errors", - "detail": "Field validations failed.", - "errors": errors, - } - ) + raise ValidationError(code="field_errors", detail="Field validations failed.", errors=errors) From 2e5dd586574600810dae67501372d254ec917722 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 14:06:35 +0200 Subject: [PATCH 360/511] snapshot --- etebase_fastapi/app.py | 6 +++++ etebase_fastapi/authentication.py | 4 +-- etebase_fastapi/test_reset_view.py | 39 ++++++++++++++++++++++++++++++ 3 files changed, 46 insertions(+), 3 deletions(-) create mode 100644 etebase_fastapi/test_reset_view.py diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index acfb42f..449059a 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -3,6 +3,8 @@ import os from django.core.wsgi import get_wsgi_application from fastapi.middleware.cors import CORSMiddleware +from django.conf import settings + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") application = get_wsgi_application() from fastapi import FastAPI, Request @@ -17,6 +19,10 @@ VERSION = "v1" BASE_PATH = f"/api/{VERSION}" app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication") app.include_router(collection_router, prefix=f"{BASE_PATH}/collection") +if settings.DEBUG: + from .test_reset_view import test_reset_view_router + + app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] ) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 287b46e..9c770d4 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -217,7 +217,6 @@ def validate_login_request( detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(validated_data.host, host_from_request) content = {"code": "wrong_host", "detail": detail} return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) - verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) try: @@ -272,7 +271,6 @@ async def change_password(data: ChangePassword, request: Request, user: User = D return Response(status_code=status.HTTP_204_NO_CONTENT) -@sync_to_async def signup_save(data: SignupIn) -> User: user_data = data.user with transaction.atomic(): @@ -309,7 +307,7 @@ def send_user_signed_up_async(user: User, request): @authentication_router.post("/signup/") async def signup(data: SignupIn): - user = await signup_save(data) + user = await sync_to_async(signup_save)(data) # XXX-TOM data = await login_response_data(user) await send_user_signed_up_async(user, None) diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py new file mode 100644 index 0000000..ee6a1c3 --- /dev/null +++ b/etebase_fastapi/test_reset_view.py @@ -0,0 +1,39 @@ +from django.conf import settings +from django.contrib.auth import get_user_model +from django.db import transaction +from django.shortcuts import get_object_or_404 +from fastapi import APIRouter, Response, status, Depends + +from django_etebase.utils import get_user_queryset +from etebase_fastapi.authentication import get_authenticated_user, SignupIn, signup_save +from etebase_fastapi.msgpack import MsgpackRoute + +test_reset_view_router = APIRouter(route_class=MsgpackRoute) +User = get_user_model() + + +@test_reset_view_router.post("/reset/") +def reset(data: SignupIn): + # Only run when in DEBUG mode! It's only used for tests + if not settings.DEBUG: + return Response("Only allowed in debug mode.", status_code=status.HTTP_400_BAD_REQUEST) + + with transaction.atomic(): + # XXX-TOM + user_queryset = get_user_queryset(User.objects.all(), None) + user = get_object_or_404(user_queryset, username=data.user.username) + # Only allow test users for extra safety + if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): + return Response("Endpoint not allowed for user.", status_code=status.HTTP_400_BAD_REQUEST) + + if hasattr(user, "userinfo"): + user.userinfo.delete() + signup_save(data) + # Delete all of the journal data for this user for a clear test env + user.collection_set.all().delete() + user.collectionmember_set.all().delete() + user.incoming_invitations.all().delete() + + # FIXME: also delete chunk files!!! + + return Response(status_code=status.HTTP_204_NO_CONTENT) From a0d1d23d2d6a1ba11a84274bc96667f3d21625cf Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 17:22:14 +0200 Subject: [PATCH 361/511] imports --- etebase_fastapi/test_reset_view.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py index ee6a1c3..ea7d8d9 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/test_reset_view.py @@ -2,10 +2,10 @@ from django.conf import settings from django.contrib.auth import get_user_model from django.db import transaction from django.shortcuts import get_object_or_404 -from fastapi import APIRouter, Response, status, Depends +from fastapi import APIRouter, Response, status from django_etebase.utils import get_user_queryset -from etebase_fastapi.authentication import get_authenticated_user, SignupIn, signup_save +from etebase_fastapi.authentication import SignupIn, signup_save from etebase_fastapi.msgpack import MsgpackRoute test_reset_view_router = APIRouter(route_class=MsgpackRoute) From 31e0e0b8323060e8265538ad96666d63724ae1f4 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 17:23:44 +0200 Subject: [PATCH 362/511] change response content to pydantic models and error handling --- etebase_fastapi/authentication.py | 115 +++++++++++++----------------- 1 file changed, 51 insertions(+), 64 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 9c770d4..9ecee4c 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -2,7 +2,6 @@ import dataclasses import typing as t from datetime import datetime from functools import cached_property -from django.core import exceptions as django_exceptions import nacl import nacl.encoding @@ -12,6 +11,7 @@ import nacl.signing from asgiref.sync import sync_to_async from django.conf import settings from django.contrib.auth import get_user_model, user_logged_out, user_logged_in +from django.core import exceptions as django_exceptions from django.db import transaction from django.utils import timezone from fastapi import APIRouter, Depends, status, Request, Response @@ -21,7 +21,6 @@ from pydantic import BaseModel from django_etebase import app_settings, models from django_etebase.exceptions import EtebaseValidationError from django_etebase.models import UserInfo -from django_etebase.serializers import UserSerializer from django_etebase.signals import user_signed_up from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry @@ -43,10 +42,16 @@ class AuthData: token: AuthToken -class LoginChallengeData(BaseModel): +class LoginChallengeIn(BaseModel): username: str +class LoginChallengeOut(BaseModel): + salt: bytes + challenge: bytes + version: int + + class LoginResponse(BaseModel): username: str challenge: bytes @@ -54,6 +59,26 @@ class LoginResponse(BaseModel): action: t.Literal["login", "changePassword"] +class UserOut(BaseModel): + pubkey: bytes + encryptedContent: bytes + + @classmethod + def from_orm(cls: t.Type["UserOut"], obj: User) -> "UserOut": + return cls(pubkey=obj.userinfo.pubkey, encryptedContent=obj.userinfo.encryptedContent) + + +class LoginOut(BaseModel): + token: str + user: UserOut + + @classmethod + def from_orm(cls: t.Type["LoginOut"], obj: User) -> "LoginOut": + token = AuthToken.objects.create(user=obj).key + user = UserOut.from_orm(obj) + return cls(token=token, user=user) + + class Authentication(BaseModel): class Config: keep_untouched = (cached_property,) @@ -145,7 +170,7 @@ def __get_login_user(username: str) -> User: raise AuthenticationFailed(code="user_not_found", detail="User not found") -async def get_login_user(challenge: LoginChallengeData) -> User: +async def get_login_user(challenge: LoginChallengeIn) -> User: user = await __get_login_user(challenge.username) return user @@ -161,7 +186,6 @@ def get_encryption_key(salt): ) -@sync_to_async def save_changed_password(data: ChangePassword, user: User): response_data = data.response_data user_info: UserInfo = user.userinfo @@ -170,24 +194,6 @@ def save_changed_password(data: ChangePassword, user: User): user_info.save() -@sync_to_async -def login_response_data(user: User): - return { - "token": AuthToken.objects.create(user=user).key, - "user": UserSerializer(user).data, - } - - -@sync_to_async -def send_user_logged_in_async(user: User, request: Request): - user_logged_in.send(sender=user.__class__, request=request, user=user) - - -@sync_to_async -def send_user_logged_out_async(user: User, request: Request): - user_logged_out.send(sender=user.__class__, request=request, user=user) - - @sync_to_async def validate_login_request( validated_data: LoginResponse, @@ -195,39 +201,26 @@ def validate_login_request( user: User, expected_action: str, host_from_request: str, -) -> t.Optional[MsgpackResponse]: - +): enc_key = get_encryption_key(bytes(user.userinfo.salt)) box = nacl.secret.SecretBox(enc_key) challenge_data = msgpack_decode(box.decrypt(validated_data.challenge)) now = int(datetime.now().timestamp()) if validated_data.action != expected_action: - content = { - "code": "wrong_action", - "detail": 'Expected "{}" but got something else'.format(challenge_sent_to_user.response), - } - return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + raise ValidationError("wrong_action", f'Expected "{challenge_sent_to_user.response}" but got something else') elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: - content = {"code": "challenge_expired", "detail": "Login challenge has expired"} - return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + raise ValidationError("challenge_expired", "Login challenge has expired") elif challenge_data["userId"] != user.id: - content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} - return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + raise ValidationError("wrong_user", "This challenge is for the wrong user") elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request: - detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(validated_data.host, host_from_request) - content = {"code": "wrong_host", "detail": detail} - return MsgpackResponse(content, status_code=status.HTTP_400_BAD_REQUEST) + raise ValidationError( + "wrong_host", f'Found wrong host name. Got: "{validated_data.host}" expected: "{host_from_request}"' + ) verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) - try: verify_key.verify(challenge_sent_to_user.response, challenge_sent_to_user.signature) except nacl.exceptions.BadSignatureError: - return MsgpackResponse( - {"code": "login_bad_signature", "detail": "Wrong password for user."}, - status_code=status.HTTP_401_UNAUTHORIZED, - ) - - return None + raise ValidationError("login_bad_signature", "Wrong password for user.", status.HTTP_401_UNAUTHORIZED) @authentication_router.post("/login_challenge/") @@ -239,35 +232,34 @@ async def login_challenge(user: User = Depends(get_login_user)): "userId": user.id, } challenge = bytes(box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder)) - return MsgpackResponse({"salt": user.userinfo.salt, "version": user.userinfo.version, "challenge": challenge}) + return MsgpackResponse( + LoginChallengeOut(salt=user.userinfo.salt, challenge=challenge, version=user.userinfo.version) + ) @authentication_router.post("/login/") async def login(data: Login, request: Request): - user = await get_login_user(LoginChallengeData(username=data.response_data.username)) + user = await get_login_user(LoginChallengeIn(username=data.response_data.username)) host = request.headers.get("Host") - bad_login_response = await validate_login_request(data.response_data, data, user, "login", host) - if bad_login_response is not None: - return bad_login_response - data = await login_response_data(user) - await send_user_logged_in_async(user, request) - return MsgpackResponse(data, status_code=status.HTTP_200_OK) + await validate_login_request(data.response_data, data, user, "login", host) + data = await sync_to_async(LoginOut.from_orm)(user) + await sync_to_async(user_logged_in.send)(sender=user.__class__, request=None, user=user) + return MsgpackResponse(content=data, status_code=status.HTTP_200_OK) @authentication_router.post("/logout/") async def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): await sync_to_async(auth_data.token.delete)() - await send_user_logged_out_async(auth_data.user, request) + # XXX-TOM + await sync_to_async(user_logged_out.send)(sender=auth_data.user.__class__, request=None, user=auth_data.user) return Response(status_code=status.HTTP_204_NO_CONTENT) @authentication_router.post("/change_password/") async def change_password(data: ChangePassword, request: Request, user: User = Depends(get_authenticated_user)): host = request.headers.get("Host") - bad_login_response = await validate_login_request(data.response_data, data, user, "changePassword", host) - if bad_login_response is not None: - return bad_login_response - await save_changed_password(data, user) + await validate_login_request(data.response_data, data, user, "changePassword", host) + await sync_to_async(save_changed_password)(data, user) return Response(status_code=status.HTTP_204_NO_CONTENT) @@ -300,15 +292,10 @@ def signup_save(data: SignupIn) -> User: return instance -@sync_to_async -def send_user_signed_up_async(user: User, request): - user_signed_up.send(sender=user.__class__, request=request, user=user) - - @authentication_router.post("/signup/") async def signup(data: SignupIn): user = await sync_to_async(signup_save)(data) # XXX-TOM - data = await login_response_data(user) - await send_user_signed_up_async(user, None) + data = await sync_to_async(LoginOut.from_orm)(user) + await sync_to_async(user_signed_up.send)(sender=user.__class__, request=None, user=user) return MsgpackResponse(content=data, status_code=status.HTTP_201_CREATED) From 4bd826b3bed6fbcbd2b5029094c38e548b38583d Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 19:08:22 +0200 Subject: [PATCH 363/511] remove uvicorn run --- etebase_fastapi/app.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 449059a..fac2a31 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -31,9 +31,3 @@ app.add_middleware( @app.exception_handler(CustomHttpException) async def custom_exception_handler(request: Request, exc: CustomHttpException): return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) - - -if __name__ == "__main__": - import uvicorn - - uvicorn.run(app, host="0.0.0.0", port=8000) From be7b830804bbb693ce9abd4e0c095f29ad10a4b9 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Fri, 25 Dec 2020 19:23:46 +0200 Subject: [PATCH 364/511] collection.py: create --- etebase_fastapi/collection.py | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 7c9aca8..ba462c9 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -1,16 +1,16 @@ import typing as t +from asgiref.sync import sync_to_async from django.contrib.auth import get_user_model from django.db.models import Q from django.db.models import QuerySet -from fastapi import APIRouter, Depends +from fastapi import APIRouter, Depends, status from pydantic import BaseModel -from asgiref.sync import sync_to_async -from django_etebase.models import Collection, Stoken, AccessLevels, CollectionMember +from django_etebase.models import Collection, AccessLevels, CollectionMember from .authentication import get_authenticated_user from .msgpack import MsgpackRoute, MsgpackResponse -from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_queryset_stoken +from .stoken_handler import filter_by_stoken_and_limit User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -75,3 +75,29 @@ async def list_multi( ) response = await list_common(queryset, user, stoken, limit) return response + + +class CollectionItemContent(BaseModel): + uid: str + meta: bytes + deleted: bool + chunks: t.List[t.List[t.Union[str, bytes]]] + + +class Item(BaseModel): + uid: str + version: int + etag: t.Optional[str] + content: CollectionItemContent + + +class CollectionItemIn(BaseModel): + collectionType: bytes + collectionKey: bytes + item: Item + + +@collection_router.post("/") +def create(data: CollectionItemIn): + # FIXME save actual item + return MsgpackResponse({}, status_code=status.HTTP_201_CREATED) From daac0c163b8b6943d89bd010b68d1c2fdae3ca9d Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sat, 26 Dec 2020 12:39:20 +0200 Subject: [PATCH 365/511] collection.py: save to db --- etebase_fastapi/collection.py | 75 ++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index ba462c9..1d876e1 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -2,13 +2,18 @@ import typing as t from asgiref.sync import sync_to_async from django.contrib.auth import get_user_model +from django.core import exceptions as django_exceptions +from django.core.files.base import ContentFile +from django.db import transaction from django.db.models import Q from django.db.models import QuerySet from fastapi import APIRouter, Depends, status from pydantic import BaseModel +from django_etebase import models from django_etebase.models import Collection, AccessLevels, CollectionMember from .authentication import get_authenticated_user +from .execptions import ValidationError from .msgpack import MsgpackRoute, MsgpackResponse from .stoken_handler import filter_by_stoken_and_limit @@ -88,6 +93,7 @@ class Item(BaseModel): uid: str version: int etag: t.Optional[str] + encryptionKey: t.Optional[bytes] content: CollectionItemContent @@ -97,7 +103,72 @@ class CollectionItemIn(BaseModel): item: Item +def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemContent): + chunks_objs = [] + + revision = models.CollectionItemRevision(**revision_data.dict(exclude={"chunks"}), item=item) + revision.validate_unique() # Verify there aren't any validation issues + + for chunk in revision_data.chunks: + uid = chunk[0] + chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() + content = chunk[1] if len(chunk) > 1 else None + # If the chunk already exists we assume it's fine. Otherwise, we upload it. + if chunk_obj is None: + if content is not None: + chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) + chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) + chunk_obj.save() + else: + raise ValidationError("chunk_no_content", "Tried to create a new chunk without content") + + chunks_objs.append(chunk_obj) + + stoken = models.Stoken.objects.create() + revision.stoken = stoken + revision.save() + + for chunk in chunks_objs: + models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) + return revision + + +def _create(data: CollectionItemIn, user: User): + with transaction.atomic(): + if data.item.etag is not None: + raise ValidationError("bad_etag", "etag is not null") + instance = models.Collection(uid=data.item.uid, owner=user) + try: + instance.validate_unique() + except django_exceptions.ValidationError: + raise ValidationError( + "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT + ) + instance.save() + + main_item = models.CollectionItem.objects.create( + uid=data.item.uid, version=data.item.version, encryptionKey=data.item.encryptionKey, collection=instance + ) + + instance.main_item = main_item + instance.save() + + # TODO + process_revisions_for_item(main_item, data.item.content) + + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=data.collectionType, owner=user) + + models.CollectionMember( + collection=instance, + stoken=models.Stoken.objects.create(), + user=user, + accessLevel=models.AccessLevels.ADMIN, + encryptionKey=data.collectionKey, + collectionType=collection_type_obj, + ).save() + + @collection_router.post("/") -def create(data: CollectionItemIn): - # FIXME save actual item +async def create(data: CollectionItemIn, user: User = Depends(get_authenticated_user)): + await sync_to_async(_create)(data, user) return MsgpackResponse({}, status_code=status.HTTP_201_CREATED) From 8d09e40b3b63e5917a345262e4a28cd728d38a05 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sat, 26 Dec 2020 18:01:55 +0200 Subject: [PATCH 366/511] rename --- etebase_fastapi/collection.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 1d876e1..b4246ca 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -97,7 +97,7 @@ class Item(BaseModel): content: CollectionItemContent -class CollectionItemIn(BaseModel): +class CollectionIn(BaseModel): collectionType: bytes collectionKey: bytes item: Item @@ -133,7 +133,7 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle return revision -def _create(data: CollectionItemIn, user: User): +def _create(data: CollectionIn, user: User): with transaction.atomic(): if data.item.etag is not None: raise ValidationError("bad_etag", "etag is not null") @@ -169,6 +169,6 @@ def _create(data: CollectionItemIn, user: User): @collection_router.post("/") -async def create(data: CollectionItemIn, user: User = Depends(get_authenticated_user)): +async def create(data: CollectionIn, user: User = Depends(get_authenticated_user)): await sync_to_async(_create)(data, user) return MsgpackResponse({}, status_code=status.HTTP_201_CREATED) From 1e60938430884fb8a8cf9d0d1710e92894d81bb4 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sat, 26 Dec 2020 18:02:29 +0200 Subject: [PATCH 367/511] rename --- etebase_fastapi/collection.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index b4246ca..9e64072 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -82,7 +82,7 @@ async def list_multi( return response -class CollectionItemContent(BaseModel): +class CollectionItemRevision(BaseModel): uid: str meta: bytes deleted: bool @@ -94,7 +94,7 @@ class Item(BaseModel): version: int etag: t.Optional[str] encryptionKey: t.Optional[bytes] - content: CollectionItemContent + content: CollectionItemRevision class CollectionIn(BaseModel): @@ -103,7 +103,7 @@ class CollectionIn(BaseModel): item: Item -def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemContent): +def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevision): chunks_objs = [] revision = models.CollectionItemRevision(**revision_data.dict(exclude={"chunks"}), item=item) From 94161943ca019d1e887ce8665e1c0e82df941eee Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sat, 26 Dec 2020 18:09:46 +0200 Subject: [PATCH 368/511] chunks type hint --- etebase_fastapi/collection.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 9e64072..77abe0f 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -86,7 +86,7 @@ class CollectionItemRevision(BaseModel): uid: str meta: bytes deleted: bool - chunks: t.List[t.List[t.Union[str, bytes]]] + chunks: t.List[t.Tuple[str, t.Optional[bytes]]] class Item(BaseModel): From 53662451a34e413457beb806564bcd98ac33e743 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sun, 27 Dec 2020 11:36:18 +0200 Subject: [PATCH 369/511] collection.py: get by uid and fixed create --- etebase_fastapi/collection.py | 115 ++++++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 32 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 77abe0f..0ceb988 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -7,8 +7,8 @@ from django.core.files.base import ContentFile from django.db import transaction from django.db.models import Q from django.db.models import QuerySet -from fastapi import APIRouter, Depends, status -from pydantic import BaseModel +from fastapi import APIRouter, Depends, status, Query +from pydantic import BaseModel, Field from django_etebase import models from django_etebase.models import Collection, AccessLevels, CollectionMember @@ -22,12 +22,57 @@ collection_router = APIRouter(route_class=MsgpackRoute) default_queryset = Collection.objects.all() +Prefetch = t.Literal["auto", "medium"] +PrefetchQuery = Query(default="auto") + + class ListMulti(BaseModel): collectionTypes: t.List[bytes] +class CollectionItemRevisionOut(BaseModel): + uid: str + meta: bytes + deleted: bool + chunks: t.List[t.Tuple[str, t.Optional[bytes]]] + + class Config: + orm_mode = True + + @classmethod + def from_orm_user( + cls: t.Type["CollectionItemRevisionOut"], obj: models.CollectionItemRevision, prefetch: Prefetch + ) -> "CollectionItemRevisionOut": + chunk_obj = obj.chunks_relation.get().chunk + if prefetch == "auto": + with open(chunk_obj.chunkFile.path, "rb") as f: + chunks = chunk_obj.uid, f.read() + else: + chunks = (chunk_obj.uid,) + return cls(uid=obj.uid, meta=obj.meta, deleted=obj.deleted, chunks=[chunks]) + + class CollectionItemOut(BaseModel): uid: str + version: int + encryptionKey: t.Optional[bytes] + etag: t.Optional[str] + content: CollectionItemRevisionOut + + class Config: + orm_mode = True + + @classmethod + def from_orm_user( + cls: t.Type["CollectionItemOut"], obj: models.CollectionItem, prefetch: Prefetch + ) -> "CollectionItemOut": + return cls( + uid=obj.uid, + version=obj.version, + encryptionKey=obj.encryptionKey, + etag=obj.etag, + content=CollectionItemRevisionOut.from_orm_user(obj.content, prefetch), + ) class CollectionOut(BaseModel): @@ -38,16 +83,17 @@ class CollectionOut(BaseModel): item: CollectionItemOut @classmethod - def from_orm_user(cls: t.Type["CollectionOut"], obj: Collection, user: User) -> "CollectionOut": + def from_orm_user(cls: t.Type["CollectionOut"], obj: Collection, user: User, prefetch: Prefetch) -> "CollectionOut": member: CollectionMember = obj.members.get(user=user) collection_type = member.collectionType - return cls( + ret = cls( collectionType=collection_type and collection_type.uid, collectionKey=member.encryptionKey, accessLevel=member.accessLevel, stoken=obj.stoken, - item=CollectionItemOut(uid=obj.main_item.uid), + item=CollectionItemOut.from_orm_user(obj.main_item, prefetch), ) + return ret class ListResponse(BaseModel): @@ -56,11 +102,26 @@ class ListResponse(BaseModel): done: bool +class ItemIn(BaseModel): + uid: str + version: int + etag: t.Optional[str] + content: CollectionItemRevisionOut + + +class CollectionIn(BaseModel): + collectionType: bytes + collectionKey: bytes + item: ItemIn + + @sync_to_async -def list_common(queryset: QuerySet, user: User, stoken: t.Optional[str], limit: int) -> MsgpackResponse: +def list_common( + queryset: QuerySet, user: User, stoken: t.Optional[str], limit: int, prefetch: Prefetch +) -> MsgpackResponse: result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, Collection.stoken_annotation) new_stoken = new_stoken_obj and new_stoken_obj.uid - data: t.List[CollectionOut] = [CollectionOut.from_orm_user(item, user) for item in queryset] + data: t.List[CollectionOut] = [CollectionOut.from_orm_user(item, user, prefetch) for item in queryset] ret = ListResponse(data=data, stoken=new_stoken, done=done) return MsgpackResponse(content=ret) @@ -71,39 +132,22 @@ def get_collection_queryset(user: User, queryset: QuerySet) -> QuerySet: @collection_router.post("/list_multi/") async def list_multi( - data: ListMulti, stoken: t.Optional[str] = None, limit: int = 50, user: User = Depends(get_authenticated_user) + data: ListMulti, + stoken: t.Optional[str] = None, + limit: int = 50, + user: User = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery, ): queryset = get_collection_queryset(user, default_queryset) # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) ) - response = await list_common(queryset, user, stoken, limit) + response = await list_common(queryset, user, stoken, limit, prefetch) return response -class CollectionItemRevision(BaseModel): - uid: str - meta: bytes - deleted: bool - chunks: t.List[t.Tuple[str, t.Optional[bytes]]] - - -class Item(BaseModel): - uid: str - version: int - etag: t.Optional[str] - encryptionKey: t.Optional[bytes] - content: CollectionItemRevision - - -class CollectionIn(BaseModel): - collectionType: bytes - collectionKey: bytes - item: Item - - -def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevision): +def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionOut): chunks_objs = [] revision = models.CollectionItemRevision(**revision_data.dict(exclude={"chunks"}), item=item) @@ -147,7 +191,7 @@ def _create(data: CollectionIn, user: User): instance.save() main_item = models.CollectionItem.objects.create( - uid=data.item.uid, version=data.item.version, encryptionKey=data.item.encryptionKey, collection=instance + uid=data.item.uid, version=data.item.version, collection=instance ) instance.main_item = main_item @@ -172,3 +216,10 @@ def _create(data: CollectionIn, user: User): async def create(data: CollectionIn, user: User = Depends(get_authenticated_user)): await sync_to_async(_create)(data, user) return MsgpackResponse({}, status_code=status.HTTP_201_CREATED) + + +@collection_router.get("/{uid}/") +def get_collection(uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery): + obj = get_collection_queryset(user, default_queryset).get(uid=uid) + ret = CollectionOut.from_orm_user(obj, user, prefetch) + return MsgpackResponse(ret) From b3c177faa63c47e99411cd56802e75ee84304bad Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sun, 27 Dec 2020 14:23:19 +0200 Subject: [PATCH 370/511] from_orm_context --- etebase_fastapi/collection.py | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 0ceb988..2b0b876 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -1,3 +1,4 @@ +import dataclasses import typing as t from asgiref.sync import sync_to_async @@ -26,6 +27,12 @@ Prefetch = t.Literal["auto", "medium"] PrefetchQuery = Query(default="auto") +@dataclasses.dataclass +class Context: + user: t.Optional[User] + prefetch: t.Optional[Prefetch] + + class ListMulti(BaseModel): collectionTypes: t.List[bytes] @@ -40,11 +47,11 @@ class CollectionItemRevisionOut(BaseModel): orm_mode = True @classmethod - def from_orm_user( - cls: t.Type["CollectionItemRevisionOut"], obj: models.CollectionItemRevision, prefetch: Prefetch + def from_orm_context( + cls: t.Type["CollectionItemRevisionOut"], obj: models.CollectionItemRevision, context: Context ) -> "CollectionItemRevisionOut": chunk_obj = obj.chunks_relation.get().chunk - if prefetch == "auto": + if context.prefetch == "auto": with open(chunk_obj.chunkFile.path, "rb") as f: chunks = chunk_obj.uid, f.read() else: @@ -63,15 +70,15 @@ class CollectionItemOut(BaseModel): orm_mode = True @classmethod - def from_orm_user( - cls: t.Type["CollectionItemOut"], obj: models.CollectionItem, prefetch: Prefetch + def from_orm_context( + cls: t.Type["CollectionItemOut"], obj: models.CollectionItem, context: Context ) -> "CollectionItemOut": return cls( uid=obj.uid, version=obj.version, encryptionKey=obj.encryptionKey, etag=obj.etag, - content=CollectionItemRevisionOut.from_orm_user(obj.content, prefetch), + content=CollectionItemRevisionOut.from_orm_context(obj.content, context), ) @@ -83,15 +90,15 @@ class CollectionOut(BaseModel): item: CollectionItemOut @classmethod - def from_orm_user(cls: t.Type["CollectionOut"], obj: Collection, user: User, prefetch: Prefetch) -> "CollectionOut": - member: CollectionMember = obj.members.get(user=user) + def from_orm_context(cls: t.Type["CollectionOut"], obj: Collection, context: Context) -> "CollectionOut": + member: CollectionMember = obj.members.get(user=context.user) collection_type = member.collectionType ret = cls( collectionType=collection_type and collection_type.uid, collectionKey=member.encryptionKey, accessLevel=member.accessLevel, stoken=obj.stoken, - item=CollectionItemOut.from_orm_user(obj.main_item, prefetch), + item=CollectionItemOut.from_orm_context(obj.main_item, context), ) return ret @@ -121,7 +128,8 @@ def list_common( ) -> MsgpackResponse: result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, Collection.stoken_annotation) new_stoken = new_stoken_obj and new_stoken_obj.uid - data: t.List[CollectionOut] = [CollectionOut.from_orm_user(item, user, prefetch) for item in queryset] + context = Context(user, prefetch) + data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in queryset] ret = ListResponse(data=data, stoken=new_stoken, done=done) return MsgpackResponse(content=ret) @@ -221,5 +229,5 @@ async def create(data: CollectionIn, user: User = Depends(get_authenticated_user @collection_router.get("/{uid}/") def get_collection(uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery): obj = get_collection_queryset(user, default_queryset).get(uid=uid) - ret = CollectionOut.from_orm_user(obj, user, prefetch) + ret = CollectionOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) From 6e4f8f9917f07b4727adb7e9eb64634ac21b586a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 14:36:16 +0200 Subject: [PATCH 371/511] Fix list_multi to return the filtered queryset. --- etebase_fastapi/collection.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 2b0b876..ff31e27 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -129,7 +129,7 @@ def list_common( result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, Collection.stoken_annotation) new_stoken = new_stoken_obj and new_stoken_obj.uid context = Context(user, prefetch) - data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in queryset] + data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] ret = ListResponse(data=data, stoken=new_stoken, done=done) return MsgpackResponse(content=ret) From a0aaf79f42450343dee7c095a9269dabd9a1d39d Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Sun, 27 Dec 2020 15:53:31 +0200 Subject: [PATCH 372/511] item_batch and item_transaction boilerplate only --- etebase_fastapi/collection.py | 52 +++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index ff31e27..093ea8a 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -8,7 +8,7 @@ from django.core.files.base import ContentFile from django.db import transaction from django.db.models import Q from django.db.models import QuerySet -from fastapi import APIRouter, Depends, status, Query +from fastapi import APIRouter, Depends, status, Query, Request from pydantic import BaseModel, Field from django_etebase import models @@ -20,7 +20,7 @@ from .stoken_handler import filter_by_stoken_and_limit User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) -default_queryset = Collection.objects.all() +default_queryset: QuerySet = Collection.objects.all() Prefetch = t.Literal["auto", "medium"] @@ -122,6 +122,19 @@ class CollectionIn(BaseModel): item: ItemIn +class ItemDepIn(BaseModel): + etag: str + uid: str + + class Config: + orm_mode = True + + +class ItemBatchIn(BaseModel): + items: t.List[ItemIn] + deps: t.Optional[ItemDepIn] + + @sync_to_async def list_common( queryset: QuerySet, user: User, stoken: t.Optional[str], limit: int, prefetch: Prefetch @@ -155,6 +168,14 @@ async def list_multi( return response +@collection_router.post("/list/") +async def collection_list( + req: Request, + user: User = Depends(get_authenticated_user), +): + pass + + def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionOut): chunks_objs = [] @@ -231,3 +252,30 @@ def get_collection(uid: str, user: User = Depends(get_authenticated_user), prefe obj = get_collection_queryset(user, default_queryset).get(uid=uid) ret = CollectionOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) + + +def item_bulk_common(data: ItemBatchIn, user: User, stoken: str, uid: str, validate_etag: bool): + queryset = get_collection_queryset(user, default_queryset) + with transaction.atomic(): # We need this for locking the collection object + collection_object = queryset.select_for_update().get(uid=uid) + if stoken is not None and stoken != collection_object.stoken: + raise ValidationError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) + + + +def item_create(): + pass # + + +@collection_router.post("/{uid}/item/transaction/") +def item_transaction( + uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) +): + item_bulk_common(data, user, stoken, uid, validate_etag=True) + + +@collection_router.post("/{uid}/item/batch/") +def item_batch( + uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) +): + item_bulk_common(data, user, stoken, uid, validate_etag=False) From 6f543751a6f511835764cf14a98b016f47cede96 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 15:25:16 +0200 Subject: [PATCH 373/511] Fix and improve typing. --- etebase_fastapi/stoken_handler.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py index c840f0e..9ea1500 100644 --- a/etebase_fastapi/stoken_handler.py +++ b/etebase_fastapi/stoken_handler.py @@ -10,7 +10,7 @@ from django_etebase.models import Stoken StokenAnnotation = t.Any -def get_stoken_obj(stoken: t.Optional[str]): +def get_stoken_obj(stoken: t.Optional[str]) -> t.Optional[Stoken]: if stoken is not None: try: return Stoken.objects.get(uid=stoken) @@ -22,7 +22,7 @@ def get_stoken_obj(stoken: t.Optional[str]): def filter_by_stoken( stoken: t.Optional[str], queryset: QuerySet, stoken_annotation: StokenAnnotation -) -> t.Tuple[QuerySet, t.Optional[str]]: +) -> t.Tuple[QuerySet, t.Optional[Stoken]]: stoken_rev = get_stoken_obj(stoken) queryset = queryset.annotate(max_stoken=stoken_annotation).order_by("max_stoken") From df855897f8a4d9fb1d1501d87b7ae3a585aa7256 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 15:26:36 +0200 Subject: [PATCH 374/511] Fix type error. --- etebase_fastapi/stoken_handler.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py index 9ea1500..fb89651 100644 --- a/etebase_fastapi/stoken_handler.py +++ b/etebase_fastapi/stoken_handler.py @@ -38,7 +38,7 @@ def get_queryset_stoken(queryset: list) -> t.Optional[Stoken]: for row in queryset: rowmaxid = getattr(row, "max_stoken") or -1 maxid = max(maxid, rowmaxid) - new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid) + new_stoken = Stoken.objects.get(id=maxid) if (maxid >= 0) else None return new_stoken or None From 9d213350e74072aca53167de278692a039d7ad55 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 15:45:29 +0200 Subject: [PATCH 375/511] exceptions.py: fix typo in filename. --- etebase_fastapi/app.py | 2 +- etebase_fastapi/authentication.py | 2 +- etebase_fastapi/collection.py | 2 +- etebase_fastapi/{execptions.py => exceptions.py} | 0 4 files changed, 3 insertions(+), 3 deletions(-) rename etebase_fastapi/{execptions.py => exceptions.py} (100%) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index fac2a31..6fbdd04 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -9,7 +9,7 @@ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") application = get_wsgi_application() from fastapi import FastAPI, Request -from .execptions import CustomHttpException +from .exceptions import CustomHttpException from .authentication import authentication_router from .collection import collection_router from .msgpack import MsgpackResponse diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 9ecee4c..51ba80d 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -26,7 +26,7 @@ from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user from django_etebase.views import msgpack_encode, msgpack_decode -from .execptions import AuthenticationFailed, transform_validation_error, ValidationError +from .exceptions import AuthenticationFailed, transform_validation_error, ValidationError from .msgpack import MsgpackResponse, MsgpackRoute User = get_user_model() diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 093ea8a..b4e9dae 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -14,7 +14,7 @@ from pydantic import BaseModel, Field from django_etebase import models from django_etebase.models import Collection, AccessLevels, CollectionMember from .authentication import get_authenticated_user -from .execptions import ValidationError +from .exceptions import ValidationError from .msgpack import MsgpackRoute, MsgpackResponse from .stoken_handler import filter_by_stoken_and_limit diff --git a/etebase_fastapi/execptions.py b/etebase_fastapi/exceptions.py similarity index 100% rename from etebase_fastapi/execptions.py rename to etebase_fastapi/exceptions.py From 249c3dc2be7dfe2015a19e5758bb9a60e58e6b9f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 16:04:33 +0200 Subject: [PATCH 376/511] Cleanup models to have common parents. --- etebase_fastapi/collection.py | 49 ++++++++++++++++------------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index b4e9dae..f687127 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -9,7 +9,7 @@ from django.db import transaction from django.db.models import Q from django.db.models import QuerySet from fastapi import APIRouter, Depends, status, Query, Request -from pydantic import BaseModel, Field +from pydantic import BaseModel from django_etebase import models from django_etebase.models import Collection, AccessLevels, CollectionMember @@ -37,7 +37,7 @@ class ListMulti(BaseModel): collectionTypes: t.List[bytes] -class CollectionItemRevisionOut(BaseModel): +class CollectionItemRevision(BaseModel): uid: str meta: bytes deleted: bool @@ -48,8 +48,8 @@ class CollectionItemRevisionOut(BaseModel): @classmethod def from_orm_context( - cls: t.Type["CollectionItemRevisionOut"], obj: models.CollectionItemRevision, context: Context - ) -> "CollectionItemRevisionOut": + cls: t.Type["CollectionItemRevision"], obj: models.CollectionItemRevision, context: Context + ) -> "CollectionItemRevision": chunk_obj = obj.chunks_relation.get().chunk if context.prefetch == "auto": with open(chunk_obj.chunkFile.path, "rb") as f: @@ -59,13 +59,14 @@ class CollectionItemRevisionOut(BaseModel): return cls(uid=obj.uid, meta=obj.meta, deleted=obj.deleted, chunks=[chunks]) -class CollectionItemOut(BaseModel): +class CollectionItemCommon(BaseModel): uid: str version: int encryptionKey: t.Optional[bytes] - etag: t.Optional[str] - content: CollectionItemRevisionOut + content: CollectionItemRevision + +class CollectionItemOut(CollectionItemCommon): class Config: orm_mode = True @@ -82,9 +83,16 @@ class CollectionItemOut(BaseModel): ) -class CollectionOut(BaseModel): - collectionKey: bytes +class CollectionItemIn(CollectionItemCommon): + etag: t.Optional[str] + + +class CollectionCommon(BaseModel): collectionType: bytes + collectionKey: bytes + + +class CollectionOut(CollectionCommon): accessLevel: AccessLevels stoken: str item: CollectionItemOut @@ -103,35 +111,23 @@ class CollectionOut(BaseModel): return ret +class CollectionIn(CollectionCommon): + item: CollectionItemIn + + class ListResponse(BaseModel): data: t.List[CollectionOut] stoken: t.Optional[str] done: bool -class ItemIn(BaseModel): - uid: str - version: int - etag: t.Optional[str] - content: CollectionItemRevisionOut - - -class CollectionIn(BaseModel): - collectionType: bytes - collectionKey: bytes - item: ItemIn - - class ItemDepIn(BaseModel): etag: str uid: str - class Config: - orm_mode = True - class ItemBatchIn(BaseModel): - items: t.List[ItemIn] + items: t.List[CollectionItemIn] deps: t.Optional[ItemDepIn] @@ -262,7 +258,6 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: str, uid: str, valid raise ValidationError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) - def item_create(): pass # From b2fe30ac26f9d6380164baa72faa720c493de354 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 16:33:34 +0200 Subject: [PATCH 377/511] Implement item_create, batch and transaction. --- etebase_fastapi/collection.py | 71 ++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index f687127..5656fb6 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -14,7 +14,7 @@ from pydantic import BaseModel from django_etebase import models from django_etebase.models import Collection, AccessLevels, CollectionMember from .authentication import get_authenticated_user -from .exceptions import ValidationError +from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse from .stoken_handler import filter_by_stoken_and_limit @@ -79,7 +79,7 @@ class CollectionItemOut(CollectionItemCommon): version=obj.version, encryptionKey=obj.encryptionKey, etag=obj.etag, - content=CollectionItemRevisionOut.from_orm_context(obj.content, context), + content=CollectionItemRevision.from_orm_context(obj.content, context), ) @@ -125,11 +125,26 @@ class ItemDepIn(BaseModel): etag: str uid: str + def validate_db(self): + item = models.CollectionItem.objects.get(uid=self.uid) + etag = self.etag + if item.etag != etag: + raise ValidationError( + "wrong_etag", + "Wrong etag. Expected {} got {}".format(item.etag, etag), + status_code=status.HTTP_409_CONFLICT, + ) + class ItemBatchIn(BaseModel): items: t.List[CollectionItemIn] deps: t.Optional[ItemDepIn] + def validate_db(self): + if self.deps is not None: + for key, _value in self.deps: + getattr(self.deps, key).validate_db() + @sync_to_async def list_common( @@ -172,7 +187,7 @@ async def collection_list( pass -def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionOut): +def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevision): chunks_objs = [] revision = models.CollectionItemRevision(**revision_data.dict(exclude={"chunks"}), item=item) @@ -250,16 +265,60 @@ def get_collection(uid: str, user: User = Depends(get_authenticated_user), prefe return MsgpackResponse(ret) -def item_bulk_common(data: ItemBatchIn, user: User, stoken: str, uid: str, validate_etag: bool): +def item_create(item_model: CollectionItemIn, validate_etag: bool): + """Function that's called when this serializer creates an item""" + etag = item_model.etag + revision_data = item_model.content + uid = item_model.uid + + Model = models.CollectionItem + + with transaction.atomic(): + instance, created = Model.objects.get_or_create( + uid=uid, defaults=item_model.dict(exclude={"uid", "etag", "content"}) + ) + cur_etag = instance.etag if not created else None + + # If we are trying to update an up to date item, abort early and consider it a success + if cur_etag == revision_data.uid: + return instance + + if validate_etag and cur_etag != etag: + raise ValidationError( + "wrong_etag", + "Wrong etag. Expected {} got {}".format(cur_etag, etag), + status_code=status.HTTP_409_CONFLICT, + ) + + if not created: + # We don't have to use select_for_update here because the unique constraint on current guards against + # the race condition. But it's a good idea because it'll lock and wait rather than fail. + current_revision = instance.revisions.filter(current=True).select_for_update().first() + current_revision.current = None + current_revision.save() + + try: + process_revisions_for_item(instance, revision_data) + except django_exceptions.ValidationError as e: + transform_validation_error("content", e) + + return instance + + +def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid: str, validate_etag: bool): queryset = get_collection_queryset(user, default_queryset) with transaction.atomic(): # We need this for locking the collection object collection_object = queryset.select_for_update().get(uid=uid) + if stoken is not None and stoken != collection_object.stoken: raise ValidationError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) + # XXX-TOM: make sure we return compatible errors + data.validate_db() + for item in data.items: + item_create(item, validate_etag) -def item_create(): - pass # + return MsgpackResponse({}) @collection_router.post("/{uid}/item/transaction/") From aa483709c33952b91cc9d75b47ef1c68b197b22e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 16:39:20 +0200 Subject: [PATCH 378/511] Fix item creation. --- etebase_fastapi/collection.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 5656fb6..8475f25 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -265,7 +265,7 @@ def get_collection(uid: str, user: User = Depends(get_authenticated_user), prefe return MsgpackResponse(ret) -def item_create(item_model: CollectionItemIn, validate_etag: bool): +def item_create(item_model: CollectionItemIn, collection: models.Collection, validate_etag: bool): """Function that's called when this serializer creates an item""" etag = item_model.etag revision_data = item_model.content @@ -275,7 +275,7 @@ def item_create(item_model: CollectionItemIn, validate_etag: bool): with transaction.atomic(): instance, created = Model.objects.get_or_create( - uid=uid, defaults=item_model.dict(exclude={"uid", "etag", "content"}) + uid=uid, collection=collection, defaults=item_model.dict(exclude={"uid", "etag", "content"}) ) cur_etag = instance.etag if not created else None @@ -316,7 +316,7 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid # XXX-TOM: make sure we return compatible errors data.validate_db() for item in data.items: - item_create(item, validate_etag) + item_create(item, collection_object, validate_etag) return MsgpackResponse({}) From 8afca6ca96c3f15807651be64b476374adeeee98 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 16:44:18 +0200 Subject: [PATCH 379/511] kwarg items: use the same naming as django_etebase. --- etebase_fastapi/collection.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 8475f25..102a7e0 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -321,15 +321,15 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return MsgpackResponse({}) -@collection_router.post("/{uid}/item/transaction/") +@collection_router.post("/{collection_uid}/item/transaction/") def item_transaction( - uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) + collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): - item_bulk_common(data, user, stoken, uid, validate_etag=True) + item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) -@collection_router.post("/{uid}/item/batch/") +@collection_router.post("/{collection_uid}/item/batch/") def item_batch( - uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) + collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): - item_bulk_common(data, user, stoken, uid, validate_etag=False) + item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) From 92f6ccbc28f99494a464f717c769b98ba56ec8c1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:02:36 +0200 Subject: [PATCH 380/511] Implement item_list and item_get. --- etebase_fastapi/collection.py | 83 ++++++++++++++++++++++++++++++++--- 1 file changed, 77 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 102a7e0..d0a9e61 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -11,16 +11,18 @@ from django.db.models import QuerySet from fastapi import APIRouter, Depends, status, Query, Request from pydantic import BaseModel +# FIXME: it's not good that some things are imported, and some are used from the model including all of the name clashes from django_etebase import models from django_etebase.models import Collection, AccessLevels, CollectionMember from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse -from .stoken_handler import filter_by_stoken_and_limit +from .stoken_handler import filter_by_stoken_and_limit, StokenAnnotation User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) default_queryset: QuerySet = Collection.objects.all() +default_item_queryset: QuerySet = models.CollectionItem.objects.all() Prefetch = t.Literal["auto", "medium"] @@ -115,12 +117,18 @@ class CollectionIn(CollectionCommon): item: CollectionItemIn -class ListResponse(BaseModel): +class CollectionListResponse(BaseModel): data: t.List[CollectionOut] stoken: t.Optional[str] done: bool +class CollectionItemListResponse(BaseModel): + data: t.List[CollectionItemOut] + stoken: t.Optional[str] + done: bool + + class ItemDepIn(BaseModel): etag: str uid: str @@ -147,14 +155,18 @@ class ItemBatchIn(BaseModel): @sync_to_async -def list_common( - queryset: QuerySet, user: User, stoken: t.Optional[str], limit: int, prefetch: Prefetch +def collection_list_common( + queryset: QuerySet, + user: User, + stoken: t.Optional[str], + limit: int, + prefetch: Prefetch, ) -> MsgpackResponse: result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, Collection.stoken_annotation) new_stoken = new_stoken_obj and new_stoken_obj.uid context = Context(user, prefetch) data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] - ret = ListResponse(data=data, stoken=new_stoken, done=done) + ret = CollectionListResponse(data=data, stoken=new_stoken, done=done) return MsgpackResponse(content=ret) @@ -162,6 +174,19 @@ def get_collection_queryset(user: User, queryset: QuerySet) -> QuerySet: return queryset.filter(members__user=user) +def get_item_queryset( + user: User, collection_uid: str, queryset: QuerySet = default_item_queryset +) -> t.Tuple[models.Collection, QuerySet]: + try: + collection = get_collection_queryset(user, Collection.objects).get(uid=collection_uid) + except Collection.DoesNotExist: + raise ValidationError("does_not_exist", "Collection does not exist", status_code=status.HTTP_404_NOT_FOUND) + # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') + queryset = queryset.filter(collection__pk=collection.pk, revisions__current=True) + + return collection, queryset + + @collection_router.post("/list_multi/") async def list_multi( data: ListMulti, @@ -175,7 +200,8 @@ async def list_multi( queryset = queryset.filter( Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) ) - response = await list_common(queryset, user, stoken, limit, prefetch) + # XXX-TOM: missing removedMemeberships + response = await collection_list_common(queryset, user, stoken, limit, prefetch) return response @@ -305,6 +331,51 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance +@collection_router.get("/{collection_uid}/item/{uid}/") +def item_get( + collection_uid: str, uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery +): + _, queryset = get_item_queryset(user, collection_uid) + obj = queryset.get(uid=uid) + ret = CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) + return MsgpackResponse(ret) + + +@sync_to_async +def item_list_common( + queryset: QuerySet, + user: User, + stoken: t.Optional[str], + limit: int, + prefetch: Prefetch, +) -> MsgpackResponse: + result, new_stoken_obj, done = filter_by_stoken_and_limit( + stoken, limit, queryset, models.CollectionItem.stoken_annotation + ) + new_stoken = new_stoken_obj and new_stoken_obj.uid + context = Context(user, prefetch) + data: t.List[CollectionItemOut] = [CollectionItemOut.from_orm_context(item, context) for item in result] + ret = CollectionItemListResponse(data=data, stoken=new_stoken, done=done) + return MsgpackResponse(content=ret) + + +@collection_router.get("/{collection_uid}/item/") +async def item_list( + collection_uid: str, + stoken: t.Optional[str] = None, + limit: int = 50, + prefetch: Prefetch = PrefetchQuery, + withCollection: bool = False, + user: User = Depends(get_authenticated_user), +): + _, queryset = await sync_to_async(get_item_queryset)(user, collection_uid) + if not withCollection: + queryset = queryset.filter(parent__isnull=True) + + response = await item_list_common(queryset, user, stoken, limit, prefetch) + return response + + def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid: str, validate_etag: bool): queryset = get_collection_queryset(user, default_queryset) with transaction.atomic(): # We need this for locking the collection object From 611c0f3b0a89088aa36afb05e4873281eb971852 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:03:17 +0200 Subject: [PATCH 381/511] Conform to naming conventions. --- etebase_fastapi/collection.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index d0a9e61..91541ce 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -285,7 +285,7 @@ async def create(data: CollectionIn, user: User = Depends(get_authenticated_user @collection_router.get("/{uid}/") -def get_collection(uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery): +def collection_get(uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery): obj = get_collection_queryset(user, default_queryset).get(uid=uid) ret = CollectionOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) From e5dbfb57460439cdd28b0d8c9656599ecbe99f03 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:07:07 +0200 Subject: [PATCH 382/511] Make the import of models more consistent. --- etebase_fastapi/collection.py | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 91541ce..6e567fd 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -11,17 +11,15 @@ from django.db.models import QuerySet from fastapi import APIRouter, Depends, status, Query, Request from pydantic import BaseModel -# FIXME: it's not good that some things are imported, and some are used from the model including all of the name clashes from django_etebase import models -from django_etebase.models import Collection, AccessLevels, CollectionMember from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse -from .stoken_handler import filter_by_stoken_and_limit, StokenAnnotation +from .stoken_handler import filter_by_stoken_and_limit User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) -default_queryset: QuerySet = Collection.objects.all() +default_queryset: QuerySet = models.Collection.objects.all() default_item_queryset: QuerySet = models.CollectionItem.objects.all() @@ -39,7 +37,7 @@ class ListMulti(BaseModel): collectionTypes: t.List[bytes] -class CollectionItemRevision(BaseModel): +class CollectionItemRevisionInOut(BaseModel): uid: str meta: bytes deleted: bool @@ -50,8 +48,8 @@ class CollectionItemRevision(BaseModel): @classmethod def from_orm_context( - cls: t.Type["CollectionItemRevision"], obj: models.CollectionItemRevision, context: Context - ) -> "CollectionItemRevision": + cls: t.Type["CollectionItemRevisionInOut"], obj: models.CollectionItemRevision, context: Context + ) -> "CollectionItemRevisionInOut": chunk_obj = obj.chunks_relation.get().chunk if context.prefetch == "auto": with open(chunk_obj.chunkFile.path, "rb") as f: @@ -65,7 +63,7 @@ class CollectionItemCommon(BaseModel): uid: str version: int encryptionKey: t.Optional[bytes] - content: CollectionItemRevision + content: CollectionItemRevisionInOut class CollectionItemOut(CollectionItemCommon): @@ -81,7 +79,7 @@ class CollectionItemOut(CollectionItemCommon): version=obj.version, encryptionKey=obj.encryptionKey, etag=obj.etag, - content=CollectionItemRevision.from_orm_context(obj.content, context), + content=CollectionItemRevisionInOut.from_orm_context(obj.content, context), ) @@ -95,12 +93,12 @@ class CollectionCommon(BaseModel): class CollectionOut(CollectionCommon): - accessLevel: AccessLevels + accessLevel: models.AccessLevels stoken: str item: CollectionItemOut @classmethod - def from_orm_context(cls: t.Type["CollectionOut"], obj: Collection, context: Context) -> "CollectionOut": + def from_orm_context(cls: t.Type["CollectionOut"], obj: models.Collection, context: Context) -> "CollectionOut": member: CollectionMember = obj.members.get(user=context.user) collection_type = member.collectionType ret = cls( @@ -162,7 +160,7 @@ def collection_list_common( limit: int, prefetch: Prefetch, ) -> MsgpackResponse: - result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, Collection.stoken_annotation) + result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, models.Collection.stoken_annotation) new_stoken = new_stoken_obj and new_stoken_obj.uid context = Context(user, prefetch) data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] @@ -178,8 +176,8 @@ def get_item_queryset( user: User, collection_uid: str, queryset: QuerySet = default_item_queryset ) -> t.Tuple[models.Collection, QuerySet]: try: - collection = get_collection_queryset(user, Collection.objects).get(uid=collection_uid) - except Collection.DoesNotExist: + collection = get_collection_queryset(user, models.Collection.objects).get(uid=collection_uid) + except models.Collection.DoesNotExist: raise ValidationError("does_not_exist", "Collection does not exist", status_code=status.HTTP_404_NOT_FOUND) # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = queryset.filter(collection__pk=collection.pk, revisions__current=True) @@ -213,7 +211,7 @@ async def collection_list( pass -def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevision): +def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionInOut): chunks_objs = [] revision = models.CollectionItemRevision(**revision_data.dict(exclude={"chunks"}), item=item) From 407ce0b7a5222900ff8d2d2c84b8e38726afcfc4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:10:59 +0200 Subject: [PATCH 383/511] Fixed collection_list_common. --- etebase_fastapi/collection.py | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 6e567fd..0f019fa 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -15,7 +15,7 @@ from django_etebase import models from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse -from .stoken_handler import filter_by_stoken_and_limit +from .stoken_handler import filter_by_stoken_and_limit, get_stoken_obj User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -115,11 +115,17 @@ class CollectionIn(CollectionCommon): item: CollectionItemIn +class RemovedMembershipOut(BaseModel): + uid: str + + class CollectionListResponse(BaseModel): data: t.List[CollectionOut] stoken: t.Optional[str] done: bool + removedMemberships: t.Optional[RemovedMembershipOut] + class CollectionItemListResponse(BaseModel): data: t.List[CollectionItemOut] @@ -164,7 +170,22 @@ def collection_list_common( new_stoken = new_stoken_obj and new_stoken_obj.uid context = Context(user, prefetch) data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] - ret = CollectionListResponse(data=data, stoken=new_stoken, done=done) + + stoken_obj = get_stoken_obj(stoken) + removedMemberships = None + if stoken_obj is not None: + # FIXME: honour limit? (the limit should be combined for data and this because of stoken) + remed_qs = models.CollectionMemberRemoved.objects.filter(user=user, stoken__id__gt=stoken_obj.id) + if not done and new_stoken_obj is not None: + # We only filter by the new_stoken if we are not done. This is because if we are done, the new stoken + # can point to the most recent collection change rather than most recent removed membership. + remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) + + remed = remed_qs.values_list("collection__uid", flat=True) + if len(remed) > 0: + removedMemberships = [{"uid": x} for x in remed] + + ret = CollectionListResponse(data=data, stoken=new_stoken, done=done, removedMemberships=removedMemberships) return MsgpackResponse(content=ret) @@ -194,13 +215,13 @@ async def list_multi( prefetch: Prefetch = PrefetchQuery, ): queryset = get_collection_queryset(user, default_queryset) + # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) ) - # XXX-TOM: missing removedMemeberships - response = await collection_list_common(queryset, user, stoken, limit, prefetch) - return response + + return await collection_list_common(queryset, user, stoken, limit, prefetch) @collection_router.post("/list/") From 7ad98b8d28a70de1074d15f17eadc201720283a2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:30:17 +0200 Subject: [PATCH 384/511] Implement is_etebase. --- etebase_fastapi/authentication.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 51ba80d..e355aa4 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -223,6 +223,11 @@ def validate_login_request( raise ValidationError("login_bad_signature", "Wrong password for user.", status.HTTP_401_UNAUTHORIZED) +@authentication_router.get("/is_etebase/") +async def is_etebase(): + return MsgpackResponse({}) + + @authentication_router.post("/login_challenge/") async def login_challenge(user: User = Depends(get_login_user)): enc_key = get_encryption_key(user.userinfo.salt) From c6c52cfe1100ef401d4941f71f7b28601473f579 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:33:01 +0200 Subject: [PATCH 385/511] Implement collection list. --- etebase_fastapi/collection.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 0f019fa..56e2180 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -166,7 +166,9 @@ def collection_list_common( limit: int, prefetch: Prefetch, ) -> MsgpackResponse: - result, new_stoken_obj, done = filter_by_stoken_and_limit(stoken, limit, queryset, models.Collection.stoken_annotation) + result, new_stoken_obj, done = filter_by_stoken_and_limit( + stoken, limit, queryset, models.Collection.stoken_annotation + ) new_stoken = new_stoken_obj and new_stoken_obj.uid context = Context(user, prefetch) data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] @@ -227,9 +229,13 @@ async def list_multi( @collection_router.post("/list/") async def collection_list( req: Request, + stoken: t.Optional[str] = None, + limit: int = 50, + prefetch: Prefetch = PrefetchQuery, user: User = Depends(get_authenticated_user), ): - pass + queryset = get_collection_queryset(user, default_queryset) + return await collection_list_common(queryset, user, stoken, limit, prefetch) def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionInOut): From a9bc08a98d59e60a929476f21bb17551bd65648b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:39:47 +0200 Subject: [PATCH 386/511] Item batch/transaction: fix return data. --- etebase_fastapi/collection.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 56e2180..3a9ca8b 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -421,11 +421,11 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid def item_transaction( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): - item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) + return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) @collection_router.post("/{collection_uid}/item/batch/") def item_batch( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): - item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) + return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) From a3ae769a2ca3d4a9e3d1f47c67df3350dfd432c7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 17:58:15 +0200 Subject: [PATCH 387/511] Implement fetch_updates. --- etebase_fastapi/collection.py | 42 ++++++++++++++++++++++++++++++- etebase_fastapi/stoken_handler.py | 2 +- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 3a9ca8b..7679ed6 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -15,7 +15,7 @@ from django_etebase import models from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse -from .stoken_handler import filter_by_stoken_and_limit, get_stoken_obj +from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -133,6 +133,11 @@ class CollectionItemListResponse(BaseModel): done: bool +class CollectionItemBulkGetIn(BaseModel): + uid: str + etag: t.Optional[str] + + class ItemDepIn(BaseModel): etag: str uid: str @@ -417,6 +422,41 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return MsgpackResponse({}) +@collection_router.post("/{collection_uid}/item/fetch_updates/") +def fetch_updates( + collection_uid: str, + data: t.List[CollectionItemBulkGetIn], + stoken: t.Optional[str] = None, + prefetch: Prefetch = PrefetchQuery, + user: User = Depends(get_authenticated_user), +): + _, queryset = get_item_queryset(user, collection_uid) + # FIXME: make configurable? + item_limit = 200 + + if len(data) > item_limit: + raise ValidationError("too_many_items", "Request has too many items.", status_code=status.HTTP_400_BAD_REQUEST) + + queryset, stoken_rev = filter_by_stoken(stoken, queryset, models.CollectionItem.stoken_annotation) + + uids, etags = zip(*[(item.uid, item.etag) for item in data]) + revs = models.CollectionItemRevision.objects.filter(uid__in=etags, current=True) + queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) + + new_stoken_obj = get_queryset_stoken(queryset) + new_stoken = new_stoken_obj and new_stoken_obj.uid + stoken = stoken_rev and getattr(stoken_rev, "uid", None) + new_stoken = new_stoken or stoken + + context = Context(user, prefetch) + ret = CollectionItemListResponse( + data=[CollectionItemOut.from_orm_context(item, context) for item in queryset], + stoken=new_stoken, + done=True, # we always return all the items, so it's always done + ) + return MsgpackResponse(ret) + + @collection_router.post("/{collection_uid}/item/transaction/") def item_transaction( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py index fb89651..a976830 100644 --- a/etebase_fastapi/stoken_handler.py +++ b/etebase_fastapi/stoken_handler.py @@ -33,7 +33,7 @@ def filter_by_stoken( return queryset, stoken_rev -def get_queryset_stoken(queryset: list) -> t.Optional[Stoken]: +def get_queryset_stoken(queryset: t.Iterable[t.Any]) -> t.Optional[Stoken]: maxid = -1 for row in queryset: rowmaxid = getattr(row, "max_stoken") or -1 From e7721e8fe52fce7667017acafee7a2c0bd8a7143 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 18:12:16 +0200 Subject: [PATCH 388/511] Fix chunk handling. --- etebase_fastapi/collection.py | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 7679ed6..229b8c4 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -50,13 +50,15 @@ class CollectionItemRevisionInOut(BaseModel): def from_orm_context( cls: t.Type["CollectionItemRevisionInOut"], obj: models.CollectionItemRevision, context: Context ) -> "CollectionItemRevisionInOut": - chunk_obj = obj.chunks_relation.get().chunk - if context.prefetch == "auto": - with open(chunk_obj.chunkFile.path, "rb") as f: - chunks = chunk_obj.uid, f.read() - else: - chunks = (chunk_obj.uid,) - return cls(uid=obj.uid, meta=obj.meta, deleted=obj.deleted, chunks=[chunks]) + chunks = [] + for chunk_relation in obj.chunks_relation.all(): + chunk_obj = chunk_relation.chunk + if context.prefetch == "auto": + with open(chunk_obj.chunkFile.path, "rb") as f: + chunks.append((chunk_obj.uid, f.read())) + else: + chunks.append((chunk_obj.uid,)) + return cls(uid=obj.uid, meta=obj.meta, deleted=obj.deleted, chunks=chunks) class CollectionItemCommon(BaseModel): From e686f016521826c8764d8851bd594cef94848a6a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 18:34:23 +0200 Subject: [PATCH 389/511] Utils: add a utility for getting objects or 404ing. --- etebase_fastapi/collection.py | 6 ++---- etebase_fastapi/utils.py | 13 +++++++++++++ 2 files changed, 15 insertions(+), 4 deletions(-) create mode 100644 etebase_fastapi/utils.py diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 229b8c4..2f8d1ed 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -16,6 +16,7 @@ from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken +from .utils import get_object_or_404 User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -205,10 +206,7 @@ def get_collection_queryset(user: User, queryset: QuerySet) -> QuerySet: def get_item_queryset( user: User, collection_uid: str, queryset: QuerySet = default_item_queryset ) -> t.Tuple[models.Collection, QuerySet]: - try: - collection = get_collection_queryset(user, models.Collection.objects).get(uid=collection_uid) - except models.Collection.DoesNotExist: - raise ValidationError("does_not_exist", "Collection does not exist", status_code=status.HTTP_404_NOT_FOUND) + collection = get_object_or_404(get_collection_queryset(user, models.Collection.objects), uid=collection_uid) # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = queryset.filter(collection__pk=collection.pk, revisions__current=True) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py new file mode 100644 index 0000000..d7f8c09 --- /dev/null +++ b/etebase_fastapi/utils.py @@ -0,0 +1,13 @@ +from fastapi import status + +from django.db.models import QuerySet +from django.core.exceptions import ObjectDoesNotExist + +from .exceptions import ValidationError + + +def get_object_or_404(queryset: QuerySet, **kwargs): + try: + return queryset.get(**kwargs) + except ObjectDoesNotExist as e: + raise ValidationError("does_not_exist", str(e), status_code=status.HTTP_404_NOT_FOUND) From 533b2787bb1ac716a9ba4c670d675432b84b3cec Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 18:34:40 +0200 Subject: [PATCH 390/511] Implement item revisions. --- etebase_fastapi/collection.py | 43 +++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 2f8d1ed..8eabf9a 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -136,6 +136,12 @@ class CollectionItemListResponse(BaseModel): done: bool +class CollectionItemRevisionListResponse(BaseModel): + data: t.List[CollectionItemRevisionInOut] + iterator: t.Optional[str] + done: bool + + class CollectionItemBulkGetIn(BaseModel): uid: str etag: t.Optional[str] @@ -422,6 +428,43 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return MsgpackResponse({}) +@collection_router.get("/{collection_uid}/item/{uid}/revision/") +def item_revisions( + collection_uid: str, + uid: str, + limit: int = 50, + iterator: t.Optional[str] = None, + prefetch: Prefetch = PrefetchQuery, + user: User = Depends(get_authenticated_user), +): + _, items = get_item_queryset(user, collection_uid) + item = get_object_or_404(items, uid=uid) + + queryset = item.revisions.order_by("-id") + + if iterator is not None: + iterator_obj = get_object_or_404(queryset, uid=iterator) + queryset = queryset.filter(id__lt=iterator_obj.id) + + result = list(queryset[: limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + context = Context(user, prefetch) + ret_data = [CollectionItemRevisionInOut.from_orm_context(revision, context) for revision in result] + iterator = ret_data[-1].uid if len(result) > 0 else None + + ret = CollectionItemRevisionListResponse( + data=ret_data, + iterator=iterator, + done=done, + ) + return MsgpackResponse(ret) + + @collection_router.post("/{collection_uid}/item/fetch_updates/") def fetch_updates( collection_uid: str, From 629a84f43243e1ee905ed985458cdd98c0192774 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 18:38:18 +0200 Subject: [PATCH 391/511] app.py: cleanup a bit. --- etebase_fastapi/app.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 6fbdd04..81772b4 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -3,10 +3,12 @@ import os from django.core.wsgi import get_wsgi_application from fastapi.middleware.cors import CORSMiddleware -from django.conf import settings - os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") application = get_wsgi_application() + +from django.conf import settings + +# Not at the top of the file because we first need to setup django from fastapi import FastAPI, Request from .exceptions import CustomHttpException From 13d4121fc275a94c6d2a53d9d894e6754c24f740 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 18:54:06 +0200 Subject: [PATCH 392/511] Move utility functions to utils. --- etebase_fastapi/collection.py | 15 ++------------- etebase_fastapi/utils.py | 17 ++++++++++++++++- 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 8eabf9a..25883f8 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -1,4 +1,3 @@ -import dataclasses import typing as t from asgiref.sync import sync_to_async @@ -8,7 +7,7 @@ from django.core.files.base import ContentFile from django.db import transaction from django.db.models import Q from django.db.models import QuerySet -from fastapi import APIRouter, Depends, status, Query, Request +from fastapi import APIRouter, Depends, status, Request from pydantic import BaseModel from django_etebase import models @@ -16,7 +15,7 @@ from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error from .msgpack import MsgpackRoute, MsgpackResponse from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken -from .utils import get_object_or_404 +from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -24,16 +23,6 @@ default_queryset: QuerySet = models.Collection.objects.all() default_item_queryset: QuerySet = models.CollectionItem.objects.all() -Prefetch = t.Literal["auto", "medium"] -PrefetchQuery = Query(default="auto") - - -@dataclasses.dataclass -class Context: - user: t.Optional[User] - prefetch: t.Optional[Prefetch] - - class ListMulti(BaseModel): collectionTypes: t.List[bytes] diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index d7f8c09..d9bef73 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -1,10 +1,25 @@ -from fastapi import status +import dataclasses +import typing as t + +from fastapi import status, Query from django.db.models import QuerySet from django.core.exceptions import ObjectDoesNotExist +from django.contrib.auth import get_user_model from .exceptions import ValidationError +User = get_user_model() + +Prefetch = t.Literal["auto", "medium"] +PrefetchQuery = Query(default="auto") + + +@dataclasses.dataclass +class Context: + user: t.Optional[User] + prefetch: t.Optional[Prefetch] + def get_object_or_404(queryset: QuerySet, **kwargs): try: From ec8c69b3f3ff0c12a31dc941722c86d7ace6d36e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 20:36:11 +0200 Subject: [PATCH 393/511] Fix a few FIXMEs. --- etebase_fastapi/authentication.py | 21 +++++++++++---------- etebase_fastapi/test_reset_view.py | 6 +++--- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index e355aa4..a8fbbfe 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -24,7 +24,7 @@ from django_etebase.models import UserInfo from django_etebase.signals import user_signed_up from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry -from django_etebase.utils import create_user +from django_etebase.utils import create_user, get_user_queryset, CallbackContext from django_etebase.views import msgpack_encode, msgpack_decode from .exceptions import AuthenticationFailed, transform_validation_error, ValidationError from .msgpack import MsgpackResponse, MsgpackRoute @@ -268,20 +268,21 @@ async def change_password(data: ChangePassword, request: Request, user: User = D return Response(status_code=status.HTTP_204_NO_CONTENT) -def signup_save(data: SignupIn) -> User: +def signup_save(data: SignupIn, request: Request) -> User: user_data = data.user with transaction.atomic(): try: - # XXX-TOM - # view = self.context.get("view", None) - # user_queryset = get_user_queryset(User.objects.all(), view) - user_queryset = User.objects.all() + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) instance = user_queryset.get(**{User.USERNAME_FIELD: user_data.username.lower()}) except User.DoesNotExist: # Create the user and save the casing the user chose as the first name try: - # XXX-TOM - instance = create_user(**user_data.dict(), password=None, first_name=user_data.username, view=None) + instance = create_user( + **user_data.dict(), + password=None, + first_name=user_data.username, + context=CallbackContext(request.path_params), + ) instance.full_clean() except EtebaseValidationError as e: raise e @@ -298,8 +299,8 @@ def signup_save(data: SignupIn) -> User: @authentication_router.post("/signup/") -async def signup(data: SignupIn): - user = await sync_to_async(signup_save)(data) +async def signup(data: SignupIn, request: Request): + user = await sync_to_async(signup_save)(data, request) # XXX-TOM data = await sync_to_async(LoginOut.from_orm)(user) await sync_to_async(user_signed_up.send)(sender=user.__class__, request=None, user=user) diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py index ea7d8d9..435a56e 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/test_reset_view.py @@ -2,7 +2,7 @@ from django.conf import settings from django.contrib.auth import get_user_model from django.db import transaction from django.shortcuts import get_object_or_404 -from fastapi import APIRouter, Response, status +from fastapi import APIRouter, Request, Response, status from django_etebase.utils import get_user_queryset from etebase_fastapi.authentication import SignupIn, signup_save @@ -13,7 +13,7 @@ User = get_user_model() @test_reset_view_router.post("/reset/") -def reset(data: SignupIn): +def reset(data: SignupIn, request: Request): # Only run when in DEBUG mode! It's only used for tests if not settings.DEBUG: return Response("Only allowed in debug mode.", status_code=status.HTTP_400_BAD_REQUEST) @@ -28,7 +28,7 @@ def reset(data: SignupIn): if hasattr(user, "userinfo"): user.userinfo.delete() - signup_save(data) + signup_save(data, request) # Delete all of the journal data for this user for a clear test env user.collection_set.all().delete() user.collectionmember_set.all().delete() From 7f90edc5114b50fa2ad414a4f2abc18f263fbdf4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:01:14 +0200 Subject: [PATCH 394/511] MsgPack: handle no content. --- etebase_fastapi/msgpack.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index 399f3d0..edffd7e 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -19,7 +19,10 @@ class MsgpackRequest(Request): class MsgpackResponse(Response): media_type = "application/msgpack" - def render(self, content: t.Any) -> bytes: + def render(self, content: t.Optional[t.Any]) -> t.Optional[bytes]: + if content is None: + return b"" + if isinstance(content, BaseModel): content = content.dict() return msgpack.packb(content, use_bin_type=True) From b70f2b74705a1ce91ad28c10f643e626b5df0459 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:08:00 +0200 Subject: [PATCH 395/511] Invitations: implement invitations endpoints. --- etebase_fastapi/app.py | 3 + etebase_fastapi/invitation.py | 229 ++++++++++++++++++++++++++++++++++ etebase_fastapi/utils.py | 7 ++ 3 files changed, 239 insertions(+) create mode 100644 etebase_fastapi/invitation.py diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 81772b4..ff50ce5 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -14,6 +14,7 @@ from fastapi import FastAPI, Request from .exceptions import CustomHttpException from .authentication import authentication_router from .collection import collection_router +from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse app = FastAPI() @@ -21,6 +22,8 @@ VERSION = "v1" BASE_PATH = f"/api/{VERSION}" app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication") app.include_router(collection_router, prefix=f"{BASE_PATH}/collection") +app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming") +app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing") if settings.DEBUG: from .test_reset_view import test_reset_view_router diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py new file mode 100644 index 0000000..077dcfd --- /dev/null +++ b/etebase_fastapi/invitation.py @@ -0,0 +1,229 @@ +import typing as t + +from django.contrib.auth import get_user_model +from django.db import transaction, IntegrityError +from django.db.models import QuerySet +from fastapi import APIRouter, Depends, status, Request +from pydantic import BaseModel + +from django_etebase import models +from django_etebase.utils import get_user_queryset, CallbackContext +from .authentication import get_authenticated_user +from .exceptions import ValidationError, PermissionDenied +from .msgpack import MsgpackRoute, MsgpackResponse +from .utils import get_object_or_404, Context, is_collection_admin + +User = get_user_model() +invitation_incoming_router = APIRouter(route_class=MsgpackRoute) +invitation_outgoing_router = APIRouter(route_class=MsgpackRoute) +default_queryset: QuerySet = models.CollectionInvitation.objects.all() + + +class UserInfoOut(BaseModel): + pubkey: bytes + + class Config: + orm_mode = True + + +class CollectionInvitationAcceptIn(BaseModel): + collectionType: bytes + encryptionKey: bytes + + +class CollectionInvitationCommon(BaseModel): + uid: str + version: int + accessLevel: models.AccessLevels + username: str + collection: str + signedEncryptionKey: bytes + + +class CollectionInvitationIn(CollectionInvitationCommon): + def validate_db(self, context: Context): + if context.user.username == self.username.lower(): + raise ValidationError("no_self_invite", "Inviting yourself is not allowed") + + +class CollectionInvitationOut(CollectionInvitationCommon): + fromUsername: str + fromPubkey: bytes + + class Config: + orm_mode = True + + @classmethod + def from_orm(cls: t.Type["CollectionInvitationOut"], obj: models.CollectionInvitation) -> "CollectionInvitationOut": + return cls( + uid=obj.uid, + version=obj.version, + accessLevel=obj.accessLevel, + username=obj.user.username, + collection=obj.collection.uid, + fromUsername=obj.fromMember.user.username, + fromPubkey=obj.fromMember.user.userinfo.pubkey, + signedEncryptionKey=obj.signedEncryptionKey, + ) + + +class InvitationListResponse(BaseModel): + data: t.List[CollectionInvitationOut] + iterator: t.Optional[str] + done: bool + + +def get_incoming_queryset(user: User, queryset=default_queryset): + return queryset.filter(user=user) + + +def get_outgoing_queryset(user: User, queryset=default_queryset): + return queryset.filter(fromMember__user=user) + + +def list_common( + queryset: QuerySet, + iterator: t.Optional[str], + limit: int, +) -> MsgpackResponse: + queryset = queryset.order_by("id") + + if iterator is not None: + iterator_obj = get_object_or_404(queryset, uid=iterator) + queryset = queryset.filter(id__gt=iterator_obj.id) + + result = list(queryset[: limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + ret_data = result + iterator = ret_data[-1].uid if len(result) > 0 else None + + ret = InvitationListResponse( + data=ret_data, + iterator=iterator, + done=done, + ) + return MsgpackResponse(ret) + + +@invitation_incoming_router.get("/", response_model=InvitationListResponse) +def incoming_list( + iterator: t.Optional[str] = None, + limit: int = 50, + user: User = Depends(get_authenticated_user), +): + return list_common(get_incoming_queryset(user), iterator, limit) + + +@invitation_incoming_router.get("/{invitation_uid}/", response_model=CollectionInvitationOut) +def incoming_get( + invitation_uid: str, + user: User = Depends(get_authenticated_user), +): + queryset = get_incoming_queryset(user) + obj = get_object_or_404(queryset, uid=invitation_uid) + ret = CollectionInvitationOut.from_orm(obj) + return MsgpackResponse(ret) + + +@invitation_incoming_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) +def incoming_delete( + invitation_uid: str, + user: User = Depends(get_authenticated_user), +): + queryset = get_incoming_queryset(user) + obj = get_object_or_404(queryset, uid=invitation_uid) + obj.delete() + + +@invitation_incoming_router.post("/{invitation_uid}/accept/", status_code=status.HTTP_201_CREATED) +def incoming_accept( + invitation_uid: str, + data: CollectionInvitationAcceptIn, + user: User = Depends(get_authenticated_user), +): + queryset = get_incoming_queryset(user) + invitation = get_object_or_404(queryset, uid=invitation_uid) + + with transaction.atomic(): + user = invitation.user + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=data.collectionType, owner=user) + + models.CollectionMember.objects.create( + collection=invitation.collection, + stoken=models.Stoken.objects.create(), + user=user, + accessLevel=invitation.accessLevel, + encryptionKey=data.encryptionKey, + collectionType=collection_type_obj, + ) + + models.CollectionMemberRemoved.objects.filter(user=invitation.user, collection=invitation.collection).delete() + + invitation.delete() + + +@invitation_outgoing_router.post("/", status_code=status.HTTP_201_CREATED) +def outgoing_create( + data: CollectionInvitationIn, + request: Request, + user: User = Depends(get_authenticated_user), +): + collection = get_object_or_404(models.Collection.objects, uid=data.collection) + to_user = get_object_or_404( + get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), username=data.username + ) + + context = Context(user, None) + data.validate_db(context) + + if not is_collection_admin(collection, user): + raise PermissionDenied("admin_access_required", "User is not an admin of this collection") + + member = collection.members.get(user=user) + + with transaction.atomic(): + try: + ret = models.CollectionInvitation.objects.create( + **data.dict(exclude={"collection", "username"}), user=to_user, fromMember=member + ) + except IntegrityError: + raise ValidationError("invitation_exists", "Invitation already exists") + + return MsgpackResponse(CollectionInvitationOut.from_orm(ret), status_code=status.HTTP_201_CREATED) + + +@invitation_outgoing_router.get("/", response_model=InvitationListResponse) +def outgoing_list( + iterator: t.Optional[str] = None, + limit: int = 50, + user: User = Depends(get_authenticated_user), +): + return list_common(get_outgoing_queryset(user), iterator, limit) + + +@invitation_outgoing_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) +def outgoing_delete( + invitation_uid: str, + user: User = Depends(get_authenticated_user), +): + queryset = get_outgoing_queryset(user) + obj = get_object_or_404(queryset, uid=invitation_uid) + obj.delete() + + +@invitation_outgoing_router.get("/fetch_user_profile/", response_model=UserInfoOut) +def outgoing_fetch_user_profile( + username: str, + request: Request, + user: User = Depends(get_authenticated_user), +): + kwargs = {User.USERNAME_FIELD: username.lower()} + user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) + user_info = get_object_or_404(models.UserInfo.objects.all(), owner=user) + ret = UserInfoOut.from_orm(user_info) + return MsgpackResponse(ret) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index d9bef73..150afe8 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -7,6 +7,8 @@ from django.db.models import QuerySet from django.core.exceptions import ObjectDoesNotExist from django.contrib.auth import get_user_model +from django_etebase.models import AccessLevels + from .exceptions import ValidationError User = get_user_model() @@ -26,3 +28,8 @@ def get_object_or_404(queryset: QuerySet, **kwargs): return queryset.get(**kwargs) except ObjectDoesNotExist as e: raise ValidationError("does_not_exist", str(e), status_code=status.HTTP_404_NOT_FOUND) + + +def is_collection_admin(collection, user): + member = collection.members.filter(user=user).first() + return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) From b5a750d6d09ee98fe3f73c9b6766458711a793a1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:41:31 +0200 Subject: [PATCH 396/511] Collection: fix removed memberships. --- etebase_fastapi/collection.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 25883f8..0af6056 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -116,7 +116,7 @@ class CollectionListResponse(BaseModel): stoken: t.Optional[str] done: bool - removedMemberships: t.Optional[RemovedMembershipOut] + removedMemberships: t.Optional[t.List[RemovedMembershipOut]] class CollectionItemListResponse(BaseModel): From 36e6d3df24628cba764559add4127ab99e57c6f6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:32:48 +0200 Subject: [PATCH 397/511] Members: add member endpoints. --- etebase_fastapi/app.py | 1 + etebase_fastapi/member.py | 83 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+) create mode 100644 etebase_fastapi/member.py diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index ff50ce5..755340c 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -14,6 +14,7 @@ from fastapi import FastAPI, Request from .exceptions import CustomHttpException from .authentication import authentication_router from .collection import collection_router +from . import member # noqa from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py new file mode 100644 index 0000000..36aa5ce --- /dev/null +++ b/etebase_fastapi/member.py @@ -0,0 +1,83 @@ +import typing as t + +from django.contrib.auth import get_user_model +from django.db.models import QuerySet +from fastapi import Depends, status +from pydantic import BaseModel + +from django_etebase import models +from .authentication import get_authenticated_user +from .msgpack import MsgpackResponse +from .utils import get_object_or_404 +from .stoken_handler import filter_by_stoken_and_limit + +from .collection import collection_router, get_collection_queryset + +User = get_user_model() +default_queryset: QuerySet = models.CollectionMember.objects.all() + + +def get_queryset(user: User, collection_uid: str, queryset=default_queryset) -> t.Tuple[models.Collection, QuerySet]: + collection = get_object_or_404(get_collection_queryset(user, models.Collection.objects), uid=collection_uid) + return collection, queryset.filter(collection=collection) + + +class CollectionMemberOut(BaseModel): + username: str + accessLevel: models.AccessLevels + + class Config: + orm_mode = True + + @classmethod + def from_orm(cls: t.Type["CollectionMemberOut"], obj: models.CollectionMember) -> "CollectionMemberOut": + return cls(username=obj.user.username, accessLevel=obj.accessLevel) + + +class MemberListResponse(BaseModel): + data: t.List[CollectionMemberOut] + iterator: t.Optional[str] + done: bool + + +@collection_router.get("/{collection_uid}/member/", response_model=MemberListResponse) +def member_list( + collection_uid: str, + iterator: t.Optional[str] = None, + limit: int = 50, + user: User = Depends(get_authenticated_user), +): + _, queryset = get_queryset(user, collection_uid) + queryset = queryset.order_by("id") + result, new_stoken_obj, done = filter_by_stoken_and_limit( + iterator, limit, queryset, models.CollectionMember.stoken_annotation + ) + new_stoken = new_stoken_obj and new_stoken_obj.uid + + ret = MemberListResponse( + data=[CollectionMemberOut.from_orm(item) for item in result], + iterator=new_stoken, + done=done, + ) + return MsgpackResponse(ret) + + +@collection_router.delete("/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT) +def member_delete( + collection_uid: str, + username: str, + user: User = Depends(get_authenticated_user), +): + _, queryset = get_queryset(user, collection_uid) + obj = get_object_or_404(queryset, user__username__iexact=username) + obj.revoke() + + +@collection_router.post("/{collection_uid}/member/leave/", status_code=status.HTTP_204_NO_CONTENT) +def member_leave( + collection_uid: str, + user: User = Depends(get_authenticated_user), +): + collection, _ = get_queryset(user, collection_uid) + obj = get_object_or_404(collection.members, user=user) + obj.revoke() From e8bd8927a01617a5a8841071de30aa0120ad4e4f Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:47:30 +0200 Subject: [PATCH 398/511] Implement modifying access level. --- etebase_fastapi/member.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 36aa5ce..2eeb365 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -1,6 +1,7 @@ import typing as t from django.contrib.auth import get_user_model +from django.db import transaction from django.db.models import QuerySet from fastapi import Depends, status from pydantic import BaseModel @@ -22,6 +23,10 @@ def get_queryset(user: User, collection_uid: str, queryset=default_queryset) -> return collection, queryset.filter(collection=collection) +class CollectionMemberModifyAccessLevelIn(BaseModel): + accessLevel: models.AccessLevels + + class CollectionMemberOut(BaseModel): username: str accessLevel: models.AccessLevels @@ -73,6 +78,24 @@ def member_delete( obj.revoke() +@collection_router.patch("/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT) +def member_patch( + collection_uid: str, + username: str, + data: CollectionMemberModifyAccessLevelIn, + user: User = Depends(get_authenticated_user), +): + _, queryset = get_queryset(user, collection_uid) + instance = get_object_or_404(queryset, user__username__iexact=username) + + with transaction.atomic(): + # We only allow updating accessLevel + if instance.accessLevel != data.accessLevel: + instance.stoken = models.Stoken.objects.create() + instance.accessLevel = data.accessLevel + instance.save() + + @collection_router.post("/{collection_uid}/member/leave/", status_code=status.HTTP_204_NO_CONTENT) def member_leave( collection_uid: str, From fa0cd01a59095bb781663876f334523ed6f40b07 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:50:34 +0200 Subject: [PATCH 399/511] Authentication: implement part of get_dashboard_url. --- etebase_fastapi/authentication.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index a8fbbfe..13af2dd 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -268,6 +268,19 @@ async def change_password(data: ChangePassword, request: Request, user: User = D return Response(status_code=status.HTTP_204_NO_CONTENT) +@authentication_router.post("/dashboard_url/") +def dashboard_url(user: User = Depends(get_authenticated_user)): + # XXX-TOM + get_dashboard_url = app_settings.DASHBOARD_URL_FUNC + if get_dashboard_url is None: + raise ValidationError("not_supported", "This server doesn't have a user dashboard.") + + ret = { + "url": get_dashboard_url(request, *args, **kwargs), + } + return MsgpackResponse(ret) + + def signup_save(data: SignupIn, request: Request) -> User: user_data = data.user with transaction.atomic(): From 403d975934072ff62ed0d147d433578b75a104e8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 21:58:58 +0200 Subject: [PATCH 400/511] Collection: fix dep handling. --- etebase_fastapi/collection.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 0af6056..5e987d8 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -137,8 +137,8 @@ class CollectionItemBulkGetIn(BaseModel): class ItemDepIn(BaseModel): - etag: str uid: str + etag: str def validate_db(self): item = models.CollectionItem.objects.get(uid=self.uid) @@ -153,12 +153,12 @@ class ItemDepIn(BaseModel): class ItemBatchIn(BaseModel): items: t.List[CollectionItemIn] - deps: t.Optional[ItemDepIn] + deps: t.Optional[t.List[ItemDepIn]] def validate_db(self): if self.deps is not None: - for key, _value in self.deps: - getattr(self.deps, key).validate_db() + for dep in self.deps: + dep.validate_db() @sync_to_async From 8160a333840aa0fe1450808395346f5d62f9793d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 22:13:36 +0200 Subject: [PATCH 401/511] Get collection queryset: remove param. --- etebase_fastapi/collection.py | 14 +++++++------- etebase_fastapi/member.py | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 5e987d8..1fc6f0a 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -194,14 +194,14 @@ def collection_list_common( return MsgpackResponse(content=ret) -def get_collection_queryset(user: User, queryset: QuerySet) -> QuerySet: - return queryset.filter(members__user=user) +def get_collection_queryset(user: User) -> QuerySet: + return default_queryset.filter(members__user=user) def get_item_queryset( user: User, collection_uid: str, queryset: QuerySet = default_item_queryset ) -> t.Tuple[models.Collection, QuerySet]: - collection = get_object_or_404(get_collection_queryset(user, models.Collection.objects), uid=collection_uid) + collection = get_object_or_404(get_collection_queryset(user), uid=collection_uid) # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') queryset = queryset.filter(collection__pk=collection.pk, revisions__current=True) @@ -216,7 +216,7 @@ async def list_multi( user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): - queryset = get_collection_queryset(user, default_queryset) + queryset = get_collection_queryset(user) # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( @@ -234,7 +234,7 @@ async def collection_list( prefetch: Prefetch = PrefetchQuery, user: User = Depends(get_authenticated_user), ): - queryset = get_collection_queryset(user, default_queryset) + queryset = get_collection_queryset(user) return await collection_list_common(queryset, user, stoken, limit, prefetch) @@ -311,7 +311,7 @@ async def create(data: CollectionIn, user: User = Depends(get_authenticated_user @collection_router.get("/{uid}/") def collection_get(uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery): - obj = get_collection_queryset(user, default_queryset).get(uid=uid) + obj = get_collection_queryset(user).get(uid=uid) ret = CollectionOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) @@ -402,7 +402,7 @@ async def item_list( def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid: str, validate_etag: bool): - queryset = get_collection_queryset(user, default_queryset) + queryset = get_collection_queryset(user) with transaction.atomic(): # We need this for locking the collection object collection_object = queryset.select_for_update().get(uid=uid) diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 2eeb365..534cad1 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -19,7 +19,7 @@ default_queryset: QuerySet = models.CollectionMember.objects.all() def get_queryset(user: User, collection_uid: str, queryset=default_queryset) -> t.Tuple[models.Collection, QuerySet]: - collection = get_object_or_404(get_collection_queryset(user, models.Collection.objects), uid=collection_uid) + collection = get_object_or_404(get_collection_queryset(user), uid=collection_uid) return collection, queryset.filter(collection=collection) From df19887af7df5a78b394ec8d8182bc9cf82bfc4c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 22:27:33 +0200 Subject: [PATCH 402/511] Use dependency injection for getting collection/item queryset. --- etebase_fastapi/collection.py | 44 +++++++++++++++++------------------ etebase_fastapi/invitation.py | 28 ++++++++++------------ etebase_fastapi/member.py | 34 +++++++++------------------ 3 files changed, 45 insertions(+), 61 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 1fc6f0a..196bb1d 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -194,18 +194,19 @@ def collection_list_common( return MsgpackResponse(content=ret) -def get_collection_queryset(user: User) -> QuerySet: +def get_collection_queryset(user: User = Depends(get_authenticated_user)) -> QuerySet: return default_queryset.filter(members__user=user) -def get_item_queryset( - user: User, collection_uid: str, queryset: QuerySet = default_item_queryset -) -> t.Tuple[models.Collection, QuerySet]: - collection = get_object_or_404(get_collection_queryset(user), uid=collection_uid) +def get_collection(collection_uid: str, queryset: QuerySet = Depends(get_collection_queryset)) -> models.Collection: + return get_object_or_404(queryset, uid=collection_uid) + + +def get_item_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') - queryset = queryset.filter(collection__pk=collection.pk, revisions__current=True) + queryset = default_item_queryset.filter(collection__pk=collection.pk, revisions__current=True) - return collection, queryset + return queryset @collection_router.post("/list_multi/") @@ -213,11 +214,10 @@ async def list_multi( data: ListMulti, stoken: t.Optional[str] = None, limit: int = 50, + queryset: QuerySet = Depends(get_collection_queryset), user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): - queryset = get_collection_queryset(user) - # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") queryset = queryset.filter( Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) @@ -228,13 +228,12 @@ async def list_multi( @collection_router.post("/list/") async def collection_list( - req: Request, stoken: t.Optional[str] = None, limit: int = 50, prefetch: Prefetch = PrefetchQuery, user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_collection_queryset), ): - queryset = get_collection_queryset(user) return await collection_list_common(queryset, user, stoken, limit, prefetch) @@ -309,9 +308,12 @@ async def create(data: CollectionIn, user: User = Depends(get_authenticated_user return MsgpackResponse({}, status_code=status.HTTP_201_CREATED) -@collection_router.get("/{uid}/") -def collection_get(uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery): - obj = get_collection_queryset(user).get(uid=uid) +@collection_router.get("/{collection_uid}/") +def collection_get( + obj: models.Collection = Depends(get_collection), + user: User = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery + ): ret = CollectionOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) @@ -358,9 +360,10 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val @collection_router.get("/{collection_uid}/item/{uid}/") def item_get( - collection_uid: str, uid: str, user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery + uid: str, + queryset: QuerySet = Depends(get_item_queryset), + user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): - _, queryset = get_item_queryset(user, collection_uid) obj = queryset.get(uid=uid) ret = CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) @@ -386,14 +389,13 @@ def item_list_common( @collection_router.get("/{collection_uid}/item/") async def item_list( - collection_uid: str, + queryset: QuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, limit: int = 50, prefetch: Prefetch = PrefetchQuery, withCollection: bool = False, user: User = Depends(get_authenticated_user), ): - _, queryset = await sync_to_async(get_item_queryset)(user, collection_uid) if not withCollection: queryset = queryset.filter(parent__isnull=True) @@ -419,14 +421,13 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid @collection_router.get("/{collection_uid}/item/{uid}/revision/") def item_revisions( - collection_uid: str, uid: str, limit: int = 50, iterator: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, user: User = Depends(get_authenticated_user), + items: QuerySet = Depends(get_item_queryset), ): - _, items = get_item_queryset(user, collection_uid) item = get_object_or_404(items, uid=uid) queryset = item.revisions.order_by("-id") @@ -456,13 +457,12 @@ def item_revisions( @collection_router.post("/{collection_uid}/item/fetch_updates/") def fetch_updates( - collection_uid: str, data: t.List[CollectionItemBulkGetIn], stoken: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_item_queryset), ): - _, queryset = get_item_queryset(user, collection_uid) # FIXME: make configurable? item_limit = 200 diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 077dcfd..cbf0554 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -73,12 +73,12 @@ class InvitationListResponse(BaseModel): done: bool -def get_incoming_queryset(user: User, queryset=default_queryset): - return queryset.filter(user=user) +def get_incoming_queryset(user: User = Depends(get_authenticated_user)): + return default_queryset.filter(user=user) -def get_outgoing_queryset(user: User, queryset=default_queryset): - return queryset.filter(fromMember__user=user) +def get_outgoing_queryset(user: User = Depends(get_authenticated_user)): + return default_queryset.filter(fromMember__user=user) def list_common( @@ -114,17 +114,16 @@ def list_common( def incoming_list( iterator: t.Optional[str] = None, limit: int = 50, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_incoming_queryset), ): - return list_common(get_incoming_queryset(user), iterator, limit) + return list_common(queryset, iterator, limit) @invitation_incoming_router.get("/{invitation_uid}/", response_model=CollectionInvitationOut) def incoming_get( invitation_uid: str, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_incoming_queryset), ): - queryset = get_incoming_queryset(user) obj = get_object_or_404(queryset, uid=invitation_uid) ret = CollectionInvitationOut.from_orm(obj) return MsgpackResponse(ret) @@ -133,9 +132,8 @@ def incoming_get( @invitation_incoming_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) def incoming_delete( invitation_uid: str, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_incoming_queryset), ): - queryset = get_incoming_queryset(user) obj = get_object_or_404(queryset, uid=invitation_uid) obj.delete() @@ -144,9 +142,8 @@ def incoming_delete( def incoming_accept( invitation_uid: str, data: CollectionInvitationAcceptIn, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_incoming_queryset), ): - queryset = get_incoming_queryset(user) invitation = get_object_or_404(queryset, uid=invitation_uid) with transaction.atomic(): @@ -201,17 +198,16 @@ def outgoing_create( def outgoing_list( iterator: t.Optional[str] = None, limit: int = 50, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_outgoing_queryset), ): - return list_common(get_outgoing_queryset(user), iterator, limit) + return list_common(queryset, iterator, limit) @invitation_outgoing_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) def outgoing_delete( invitation_uid: str, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_outgoing_queryset), ): - queryset = get_outgoing_queryset(user) obj = get_object_or_404(queryset, uid=invitation_uid) obj.delete() diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 534cad1..a491490 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -12,15 +12,18 @@ from .msgpack import MsgpackResponse from .utils import get_object_or_404 from .stoken_handler import filter_by_stoken_and_limit -from .collection import collection_router, get_collection_queryset +from .collection import collection_router, get_collection User = get_user_model() default_queryset: QuerySet = models.CollectionMember.objects.all() -def get_queryset(user: User, collection_uid: str, queryset=default_queryset) -> t.Tuple[models.Collection, QuerySet]: - collection = get_object_or_404(get_collection_queryset(user), uid=collection_uid) - return collection, queryset.filter(collection=collection) +def get_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: + return default_queryset.filter(collection=collection) + + +def get_member(username: str, queryset: QuerySet = Depends(get_queryset)) -> QuerySet: + return get_object_or_404(queryset, user__username__iexact=username) class CollectionMemberModifyAccessLevelIn(BaseModel): @@ -47,12 +50,10 @@ class MemberListResponse(BaseModel): @collection_router.get("/{collection_uid}/member/", response_model=MemberListResponse) def member_list( - collection_uid: str, iterator: t.Optional[str] = None, limit: int = 50, - user: User = Depends(get_authenticated_user), + queryset: QuerySet = Depends(get_queryset), ): - _, queryset = get_queryset(user, collection_uid) queryset = queryset.order_by("id") result, new_stoken_obj, done = filter_by_stoken_and_limit( iterator, limit, queryset, models.CollectionMember.stoken_annotation @@ -69,25 +70,16 @@ def member_list( @collection_router.delete("/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT) def member_delete( - collection_uid: str, - username: str, - user: User = Depends(get_authenticated_user), + obj: models.CollectionMember = Depends(get_member), ): - _, queryset = get_queryset(user, collection_uid) - obj = get_object_or_404(queryset, user__username__iexact=username) obj.revoke() @collection_router.patch("/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT) def member_patch( - collection_uid: str, - username: str, data: CollectionMemberModifyAccessLevelIn, - user: User = Depends(get_authenticated_user), + instance: models.CollectionMember = Depends(get_member), ): - _, queryset = get_queryset(user, collection_uid) - instance = get_object_or_404(queryset, user__username__iexact=username) - with transaction.atomic(): # We only allow updating accessLevel if instance.accessLevel != data.accessLevel: @@ -97,10 +89,6 @@ def member_patch( @collection_router.post("/{collection_uid}/member/leave/", status_code=status.HTTP_204_NO_CONTENT) -def member_leave( - collection_uid: str, - user: User = Depends(get_authenticated_user), -): - collection, _ = get_queryset(user, collection_uid) +def member_leave(user: User = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection)): obj = get_object_or_404(collection.members, user=user) obj.revoke() From c7b8b0373a171e114213fa0f5337ac571ea19e3c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 22:56:23 +0200 Subject: [PATCH 403/511] Add permissions. --- etebase_fastapi/collection.py | 28 ++++++++++++++++++++++++---- etebase_fastapi/member.py | 23 ++++++++++++++++++----- 2 files changed, 42 insertions(+), 9 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 196bb1d..13c53be 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -12,10 +12,10 @@ from pydantic import BaseModel from django_etebase import models from .authentication import get_authenticated_user -from .exceptions import ValidationError, transform_validation_error +from .exceptions import ValidationError, transform_validation_error, PermissionDenied from .msgpack import MsgpackRoute, MsgpackResponse from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken -from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery +from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute) @@ -209,6 +209,26 @@ def get_item_queryset(collection: models.Collection = Depends(get_collection)) - return queryset +# permissions + + +def verify_collection_admin( + collection: models.Collection = Depends(get_collection), user: User = Depends(get_authenticated_user) +): + if not is_collection_admin(collection, user): + raise PermissionDenied("admin_access_required", "Only collection admins can perform this operation.") + + +def has_write_access( + collection: models.Collection = Depends(get_collection), user: User = Depends(get_authenticated_user) +): + member = collection.members.get(user=user) + if member.accessLevel == models.AccessLevels.READ_ONLY: + raise PermissionDenied("no_write_access", "You need write access to write to this collection") + + +# paths + @collection_router.post("/list_multi/") async def list_multi( data: ListMulti, @@ -489,14 +509,14 @@ def fetch_updates( return MsgpackResponse(ret) -@collection_router.post("/{collection_uid}/item/transaction/") +@collection_router.post("/{collection_uid}/item/transaction/", dependencies=[Depends(has_write_access)]) def item_transaction( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) -@collection_router.post("/{collection_uid}/item/batch/") +@collection_router.post("/{collection_uid}/item/batch/", dependencies=[Depends(has_write_access)]) def item_batch( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index a491490..f3c77e5 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -12,7 +12,7 @@ from .msgpack import MsgpackResponse from .utils import get_object_or_404 from .stoken_handler import filter_by_stoken_and_limit -from .collection import collection_router, get_collection +from .collection import collection_router, get_collection, verify_collection_admin User = get_user_model() default_queryset: QuerySet = models.CollectionMember.objects.all() @@ -48,7 +48,9 @@ class MemberListResponse(BaseModel): done: bool -@collection_router.get("/{collection_uid}/member/", response_model=MemberListResponse) +@collection_router.get( + "/{collection_uid}/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin)] +) def member_list( iterator: t.Optional[str] = None, limit: int = 50, @@ -68,14 +70,22 @@ def member_list( return MsgpackResponse(ret) -@collection_router.delete("/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT) +@collection_router.delete( + "/{collection_uid}/member/{username}/", + status_code=status.HTTP_204_NO_CONTENT, + dependencies=[Depends(verify_collection_admin)], +) def member_delete( obj: models.CollectionMember = Depends(get_member), ): obj.revoke() -@collection_router.patch("/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT) +@collection_router.patch( + "/{collection_uid}/member/{username}/", + status_code=status.HTTP_204_NO_CONTENT, + dependencies=[Depends(verify_collection_admin)], +) def member_patch( data: CollectionMemberModifyAccessLevelIn, instance: models.CollectionMember = Depends(get_member), @@ -88,7 +98,10 @@ def member_patch( instance.save() -@collection_router.post("/{collection_uid}/member/leave/", status_code=status.HTTP_204_NO_CONTENT) +@collection_router.post( + "/{collection_uid}/member/leave/", + status_code=status.HTTP_204_NO_CONTENT, +) def member_leave(user: User = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection)): obj = get_object_or_404(collection.members, user=user) obj.revoke() From 1c8684ee9280856d20d48ca742018dc34ac79995 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 23:06:25 +0200 Subject: [PATCH 404/511] Fix a FIXME. --- etebase_fastapi/test_reset_view.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py index 435a56e..73a803e 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/test_reset_view.py @@ -4,7 +4,7 @@ from django.db import transaction from django.shortcuts import get_object_or_404 from fastapi import APIRouter, Request, Response, status -from django_etebase.utils import get_user_queryset +from django_etebase.utils import get_user_queryset, CallbackContext from etebase_fastapi.authentication import SignupIn, signup_save from etebase_fastapi.msgpack import MsgpackRoute @@ -19,8 +19,7 @@ def reset(data: SignupIn, request: Request): return Response("Only allowed in debug mode.", status_code=status.HTTP_400_BAD_REQUEST) with transaction.atomic(): - # XXX-TOM - user_queryset = get_user_queryset(User.objects.all(), None) + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) user = get_object_or_404(user_queryset, username=data.user.username) # Only allow test users for extra safety if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): From d63c34693f4a88717353f7e708f8fed58bfe1b48 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 27 Dec 2020 23:11:12 +0200 Subject: [PATCH 405/511] Change all item_uids to be called item_uids. --- etebase_fastapi/collection.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 13c53be..33f1d64 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -378,13 +378,13 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance -@collection_router.get("/{collection_uid}/item/{uid}/") +@collection_router.get("/{collection_uid}/item/{item_uid}/") def item_get( - uid: str, + item_uid: str, queryset: QuerySet = Depends(get_item_queryset), user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): - obj = queryset.get(uid=uid) + obj = queryset.get(uid=item_uid) ret = CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) return MsgpackResponse(ret) @@ -439,16 +439,16 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return MsgpackResponse({}) -@collection_router.get("/{collection_uid}/item/{uid}/revision/") +@collection_router.get("/{collection_uid}/item/{item_uid}/revision/") def item_revisions( - uid: str, + item_uid: str, limit: int = 50, iterator: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, user: User = Depends(get_authenticated_user), items: QuerySet = Depends(get_item_queryset), ): - item = get_object_or_404(items, uid=uid) + item = get_object_or_404(items, uid=item_uid) queryset = item.revisions.order_by("-id") From 15988235f27673398416b244fd40aa39009a89f1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 08:42:48 +0200 Subject: [PATCH 406/511] Exclude unset fields so fix removedMemberships return value. --- etebase_fastapi/collection.py | 6 +++--- etebase_fastapi/msgpack.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 33f1d64..1c5ca14 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -176,8 +176,9 @@ def collection_list_common( context = Context(user, prefetch) data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] + ret = CollectionListResponse(data=data, stoken=new_stoken, done=done) + stoken_obj = get_stoken_obj(stoken) - removedMemberships = None if stoken_obj is not None: # FIXME: honour limit? (the limit should be combined for data and this because of stoken) remed_qs = models.CollectionMemberRemoved.objects.filter(user=user, stoken__id__gt=stoken_obj.id) @@ -188,9 +189,8 @@ def collection_list_common( remed = remed_qs.values_list("collection__uid", flat=True) if len(remed) > 0: - removedMemberships = [{"uid": x} for x in remed] + ret.removedMemberships = [{"uid": x} for x in remed] - ret = CollectionListResponse(data=data, stoken=new_stoken, done=done, removedMemberships=removedMemberships) return MsgpackResponse(content=ret) diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index edffd7e..0c5cc30 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -24,7 +24,7 @@ class MsgpackResponse(Response): return b"" if isinstance(content, BaseModel): - content = content.dict() + content = content.dict(exclude_unset=True) return msgpack.packb(content, use_bin_type=True) From 9f26ecf27682b3a0cbea85c10f23bfccdcdacc73 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 09:04:45 +0200 Subject: [PATCH 407/511] API: add documentation tags. --- etebase_fastapi/app.py | 6 ++++-- etebase_fastapi/authentication.py | 2 +- etebase_fastapi/collection.py | 17 +++++++++-------- etebase_fastapi/invitation.py | 4 ++-- etebase_fastapi/member.py | 15 ++++++++------- etebase_fastapi/test_reset_view.py | 2 +- 6 files changed, 25 insertions(+), 21 deletions(-) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 755340c..ffea2fb 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -13,8 +13,8 @@ from fastapi import FastAPI, Request from .exceptions import CustomHttpException from .authentication import authentication_router -from .collection import collection_router -from . import member # noqa +from .collection import collection_router, item_router +from .member import member_router from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse @@ -23,6 +23,8 @@ VERSION = "v1" BASE_PATH = f"/api/{VERSION}" app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication") app.include_router(collection_router, prefix=f"{BASE_PATH}/collection") +app.include_router(item_router, prefix=f"{BASE_PATH}/collection") +app.include_router(member_router, prefix=f"{BASE_PATH}/collection") app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming") app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing") if settings.DEBUG: diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 13af2dd..f04753b 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -33,7 +33,7 @@ User = get_user_model() token_scheme = APIKeyHeader(name="Authorization") AUTO_REFRESH = True MIN_REFRESH_INTERVAL = 60 -authentication_router = APIRouter(route_class=MsgpackRoute) +authentication_router = APIRouter(route_class=MsgpackRoute, tags=["authentication"]) @dataclasses.dataclass(frozen=True) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 1c5ca14..ead113a 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -7,7 +7,7 @@ from django.core.files.base import ContentFile from django.db import transaction from django.db.models import Q from django.db.models import QuerySet -from fastapi import APIRouter, Depends, status, Request +from fastapi import APIRouter, Depends, status from pydantic import BaseModel from django_etebase import models @@ -18,7 +18,8 @@ from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_st from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin User = get_user_model() -collection_router = APIRouter(route_class=MsgpackRoute) +collection_router = APIRouter(route_class=MsgpackRoute, tags=["collection"]) +item_router = APIRouter(route_class=MsgpackRoute, tags=["item"]) default_queryset: QuerySet = models.Collection.objects.all() default_item_queryset: QuerySet = models.CollectionItem.objects.all() @@ -378,7 +379,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance -@collection_router.get("/{collection_uid}/item/{item_uid}/") +@item_router.get("/{collection_uid}/item/{item_uid}/") def item_get( item_uid: str, queryset: QuerySet = Depends(get_item_queryset), @@ -407,7 +408,7 @@ def item_list_common( return MsgpackResponse(content=ret) -@collection_router.get("/{collection_uid}/item/") +@item_router.get("/{collection_uid}/item/") async def item_list( queryset: QuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, @@ -439,7 +440,7 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return MsgpackResponse({}) -@collection_router.get("/{collection_uid}/item/{item_uid}/revision/") +@item_router.get("/{collection_uid}/item/{item_uid}/revision/") def item_revisions( item_uid: str, limit: int = 50, @@ -475,7 +476,7 @@ def item_revisions( return MsgpackResponse(ret) -@collection_router.post("/{collection_uid}/item/fetch_updates/") +@item_router.post("/{collection_uid}/item/fetch_updates/") def fetch_updates( data: t.List[CollectionItemBulkGetIn], stoken: t.Optional[str] = None, @@ -509,14 +510,14 @@ def fetch_updates( return MsgpackResponse(ret) -@collection_router.post("/{collection_uid}/item/transaction/", dependencies=[Depends(has_write_access)]) +@item_router.post("/{collection_uid}/item/transaction/", dependencies=[Depends(has_write_access)]) def item_transaction( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) -@collection_router.post("/{collection_uid}/item/batch/", dependencies=[Depends(has_write_access)]) +@item_router.post("/{collection_uid}/item/batch/", dependencies=[Depends(has_write_access)]) def item_batch( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index cbf0554..9b166ee 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -14,8 +14,8 @@ from .msgpack import MsgpackRoute, MsgpackResponse from .utils import get_object_or_404, Context, is_collection_admin User = get_user_model() -invitation_incoming_router = APIRouter(route_class=MsgpackRoute) -invitation_outgoing_router = APIRouter(route_class=MsgpackRoute) +invitation_incoming_router = APIRouter(route_class=MsgpackRoute, tags=["incoming invitation"]) +invitation_outgoing_router = APIRouter(route_class=MsgpackRoute, tags=["outgoing invitation"]) default_queryset: QuerySet = models.CollectionInvitation.objects.all() diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index f3c77e5..af349d1 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -3,18 +3,19 @@ import typing as t from django.contrib.auth import get_user_model from django.db import transaction from django.db.models import QuerySet -from fastapi import Depends, status +from fastapi import APIRouter, Depends, status from pydantic import BaseModel from django_etebase import models from .authentication import get_authenticated_user -from .msgpack import MsgpackResponse +from .msgpack import MsgpackRoute, MsgpackResponse from .utils import get_object_or_404 from .stoken_handler import filter_by_stoken_and_limit -from .collection import collection_router, get_collection, verify_collection_admin +from .collection import get_collection, verify_collection_admin User = get_user_model() +member_router = APIRouter(route_class=MsgpackRoute, tags=["member"]) default_queryset: QuerySet = models.CollectionMember.objects.all() @@ -48,7 +49,7 @@ class MemberListResponse(BaseModel): done: bool -@collection_router.get( +@member_router.get( "/{collection_uid}/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin)] ) def member_list( @@ -70,7 +71,7 @@ def member_list( return MsgpackResponse(ret) -@collection_router.delete( +@member_router.delete( "/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=[Depends(verify_collection_admin)], @@ -81,7 +82,7 @@ def member_delete( obj.revoke() -@collection_router.patch( +@member_router.patch( "/{collection_uid}/member/{username}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=[Depends(verify_collection_admin)], @@ -98,7 +99,7 @@ def member_patch( instance.save() -@collection_router.post( +@member_router.post( "/{collection_uid}/member/leave/", status_code=status.HTTP_204_NO_CONTENT, ) diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py index 73a803e..f21fd84 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/test_reset_view.py @@ -8,7 +8,7 @@ from django_etebase.utils import get_user_queryset, CallbackContext from etebase_fastapi.authentication import SignupIn, signup_save from etebase_fastapi.msgpack import MsgpackRoute -test_reset_view_router = APIRouter(route_class=MsgpackRoute) +test_reset_view_router = APIRouter(route_class=MsgpackRoute, tags=["test helpers"]) User = get_user_model() From ee4e7cf498657ee78d3dbac6f8078085b2abc64d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 09:11:35 +0200 Subject: [PATCH 408/511] Unify the nested router prefix. --- etebase_fastapi/app.py | 5 +++-- etebase_fastapi/collection.py | 12 ++++++------ etebase_fastapi/member.py | 10 ++++------ 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index ffea2fb..a8b12d0 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -21,10 +21,11 @@ from .msgpack import MsgpackResponse app = FastAPI() VERSION = "v1" BASE_PATH = f"/api/{VERSION}" +COLLECTION_UID_MARKER = "{collection_uid}" app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication") app.include_router(collection_router, prefix=f"{BASE_PATH}/collection") -app.include_router(item_router, prefix=f"{BASE_PATH}/collection") -app.include_router(member_router, prefix=f"{BASE_PATH}/collection") +app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}") +app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}") app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming") app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing") if settings.DEBUG: diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index ead113a..c0efed1 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -379,7 +379,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance -@item_router.get("/{collection_uid}/item/{item_uid}/") +@item_router.get("/item/{item_uid}/") def item_get( item_uid: str, queryset: QuerySet = Depends(get_item_queryset), @@ -408,7 +408,7 @@ def item_list_common( return MsgpackResponse(content=ret) -@item_router.get("/{collection_uid}/item/") +@item_router.get("/item/") async def item_list( queryset: QuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, @@ -440,7 +440,7 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return MsgpackResponse({}) -@item_router.get("/{collection_uid}/item/{item_uid}/revision/") +@item_router.get("/item/{item_uid}/revision/") def item_revisions( item_uid: str, limit: int = 50, @@ -476,7 +476,7 @@ def item_revisions( return MsgpackResponse(ret) -@item_router.post("/{collection_uid}/item/fetch_updates/") +@item_router.post("/item/fetch_updates/") def fetch_updates( data: t.List[CollectionItemBulkGetIn], stoken: t.Optional[str] = None, @@ -510,14 +510,14 @@ def fetch_updates( return MsgpackResponse(ret) -@item_router.post("/{collection_uid}/item/transaction/", dependencies=[Depends(has_write_access)]) +@item_router.post("/item/transaction/", dependencies=[Depends(has_write_access)]) def item_transaction( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) -@item_router.post("/{collection_uid}/item/batch/", dependencies=[Depends(has_write_access)]) +@item_router.post("/item/batch/", dependencies=[Depends(has_write_access)]) def item_batch( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index af349d1..2c9b631 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -49,9 +49,7 @@ class MemberListResponse(BaseModel): done: bool -@member_router.get( - "/{collection_uid}/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin)] -) +@member_router.get("/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin)]) def member_list( iterator: t.Optional[str] = None, limit: int = 50, @@ -72,7 +70,7 @@ def member_list( @member_router.delete( - "/{collection_uid}/member/{username}/", + "/member/{username}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=[Depends(verify_collection_admin)], ) @@ -83,7 +81,7 @@ def member_delete( @member_router.patch( - "/{collection_uid}/member/{username}/", + "/member/{username}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=[Depends(verify_collection_admin)], ) @@ -100,7 +98,7 @@ def member_patch( @member_router.post( - "/{collection_uid}/member/leave/", + "/member/leave/", status_code=status.HTTP_204_NO_CONTENT, ) def member_leave(user: User = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection)): From 80d69a566325755107d810c6fb6c67147f32c9cc Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 09:25:28 +0200 Subject: [PATCH 409/511] Fix collection list and how we return API responses. --- etebase_fastapi/authentication.py | 28 +++++++++---------- etebase_fastapi/collection.py | 45 +++++++++++++------------------ etebase_fastapi/invitation.py | 18 +++++-------- etebase_fastapi/member.py | 8 +++--- etebase_fastapi/msgpack.py | 2 +- etebase_fastapi/utils.py | 8 ++++++ 6 files changed, 49 insertions(+), 60 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index f04753b..13a8884 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -16,7 +16,6 @@ from django.db import transaction from django.utils import timezone from fastapi import APIRouter, Depends, status, Request, Response from fastapi.security import APIKeyHeader -from pydantic import BaseModel from django_etebase import app_settings, models from django_etebase.exceptions import EtebaseValidationError @@ -27,7 +26,8 @@ from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user, get_user_queryset, CallbackContext from django_etebase.views import msgpack_encode, msgpack_decode from .exceptions import AuthenticationFailed, transform_validation_error, ValidationError -from .msgpack import MsgpackResponse, MsgpackRoute +from .msgpack import MsgpackRoute +from .utils import BaseModel User = get_user_model() token_scheme = APIKeyHeader(name="Authorization") @@ -225,10 +225,10 @@ def validate_login_request( @authentication_router.get("/is_etebase/") async def is_etebase(): - return MsgpackResponse({}) + pass -@authentication_router.post("/login_challenge/") +@authentication_router.post("/login_challenge/", response_model=LoginChallengeOut) async def login_challenge(user: User = Depends(get_login_user)): enc_key = get_encryption_key(user.userinfo.salt) box = nacl.secret.SecretBox(enc_key) @@ -237,35 +237,31 @@ async def login_challenge(user: User = Depends(get_login_user)): "userId": user.id, } challenge = bytes(box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder)) - return MsgpackResponse( - LoginChallengeOut(salt=user.userinfo.salt, challenge=challenge, version=user.userinfo.version) - ) + return LoginChallengeOut(salt=user.userinfo.salt, challenge=challenge, version=user.userinfo.version) -@authentication_router.post("/login/") +@authentication_router.post("/login/", response_model=LoginOut) async def login(data: Login, request: Request): user = await get_login_user(LoginChallengeIn(username=data.response_data.username)) host = request.headers.get("Host") await validate_login_request(data.response_data, data, user, "login", host) data = await sync_to_async(LoginOut.from_orm)(user) await sync_to_async(user_logged_in.send)(sender=user.__class__, request=None, user=user) - return MsgpackResponse(content=data, status_code=status.HTTP_200_OK) + return data -@authentication_router.post("/logout/") +@authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT) async def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): await sync_to_async(auth_data.token.delete)() # XXX-TOM await sync_to_async(user_logged_out.send)(sender=auth_data.user.__class__, request=None, user=auth_data.user) - return Response(status_code=status.HTTP_204_NO_CONTENT) -@authentication_router.post("/change_password/") +@authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT) async def change_password(data: ChangePassword, request: Request, user: User = Depends(get_authenticated_user)): host = request.headers.get("Host") await validate_login_request(data.response_data, data, user, "changePassword", host) await sync_to_async(save_changed_password)(data, user) - return Response(status_code=status.HTTP_204_NO_CONTENT) @authentication_router.post("/dashboard_url/") @@ -278,7 +274,7 @@ def dashboard_url(user: User = Depends(get_authenticated_user)): ret = { "url": get_dashboard_url(request, *args, **kwargs), } - return MsgpackResponse(ret) + return ret def signup_save(data: SignupIn, request: Request) -> User: @@ -311,10 +307,10 @@ def signup_save(data: SignupIn, request: Request) -> User: return instance -@authentication_router.post("/signup/") +@authentication_router.post("/signup/", response_model=LoginOut, status_code=status.HTTP_201_CREATED) async def signup(data: SignupIn, request: Request): user = await sync_to_async(signup_save)(data, request) # XXX-TOM data = await sync_to_async(LoginOut.from_orm)(user) await sync_to_async(user_signed_up.send)(sender=user.__class__, request=None, user=user) - return MsgpackResponse(content=data, status_code=status.HTTP_201_CREATED) + return data diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index c0efed1..993d144 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -8,14 +8,13 @@ from django.db import transaction from django.db.models import Q from django.db.models import QuerySet from fastapi import APIRouter, Depends, status -from pydantic import BaseModel from django_etebase import models from .authentication import get_authenticated_user from .exceptions import ValidationError, transform_validation_error, PermissionDenied -from .msgpack import MsgpackRoute, MsgpackResponse +from .msgpack import MsgpackRoute from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken -from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin +from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin, BaseModel User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute, tags=["collection"]) @@ -169,7 +168,7 @@ def collection_list_common( stoken: t.Optional[str], limit: int, prefetch: Prefetch, -) -> MsgpackResponse: +) -> CollectionListResponse: result, new_stoken_obj, done = filter_by_stoken_and_limit( stoken, limit, queryset, models.Collection.stoken_annotation ) @@ -192,7 +191,7 @@ def collection_list_common( if len(remed) > 0: ret.removedMemberships = [{"uid": x} for x in remed] - return MsgpackResponse(content=ret) + return ret def get_collection_queryset(user: User = Depends(get_authenticated_user)) -> QuerySet: @@ -230,7 +229,7 @@ def has_write_access( # paths -@collection_router.post("/list_multi/") +@collection_router.post("/list_multi/", response_model=CollectionListResponse, response_model_exclude_unset=True) async def list_multi( data: ListMulti, stoken: t.Optional[str] = None, @@ -247,7 +246,7 @@ async def list_multi( return await collection_list_common(queryset, user, stoken, limit, prefetch) -@collection_router.post("/list/") +@collection_router.get("/", response_model=CollectionListResponse) async def collection_list( stoken: t.Optional[str] = None, limit: int = 50, @@ -323,20 +322,18 @@ def _create(data: CollectionIn, user: User): ).save() -@collection_router.post("/") +@collection_router.post("/", status_code=status.HTTP_201_CREATED) async def create(data: CollectionIn, user: User = Depends(get_authenticated_user)): await sync_to_async(_create)(data, user) - return MsgpackResponse({}, status_code=status.HTTP_201_CREATED) -@collection_router.get("/{collection_uid}/") +@collection_router.get("/{collection_uid}/", response_model=CollectionOut) def collection_get( obj: models.Collection = Depends(get_collection), user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery ): - ret = CollectionOut.from_orm_context(obj, Context(user, prefetch)) - return MsgpackResponse(ret) + return CollectionOut.from_orm_context(obj, Context(user, prefetch)) def item_create(item_model: CollectionItemIn, collection: models.Collection, validate_etag: bool): @@ -379,15 +376,14 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance -@item_router.get("/item/{item_uid}/") +@item_router.get("/item/{item_uid}/", response_model=CollectionItemOut) def item_get( item_uid: str, queryset: QuerySet = Depends(get_item_queryset), user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): obj = queryset.get(uid=item_uid) - ret = CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) - return MsgpackResponse(ret) + return CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) @sync_to_async @@ -397,18 +393,17 @@ def item_list_common( stoken: t.Optional[str], limit: int, prefetch: Prefetch, -) -> MsgpackResponse: +) -> CollectionItemListResponse: result, new_stoken_obj, done = filter_by_stoken_and_limit( stoken, limit, queryset, models.CollectionItem.stoken_annotation ) new_stoken = new_stoken_obj and new_stoken_obj.uid context = Context(user, prefetch) data: t.List[CollectionItemOut] = [CollectionItemOut.from_orm_context(item, context) for item in result] - ret = CollectionItemListResponse(data=data, stoken=new_stoken, done=done) - return MsgpackResponse(content=ret) + return CollectionItemListResponse(data=data, stoken=new_stoken, done=done) -@item_router.get("/item/") +@item_router.get("/item/", response_model=CollectionItemListResponse) async def item_list( queryset: QuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, @@ -437,10 +432,10 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid for item in data.items: item_create(item, collection_object, validate_etag) - return MsgpackResponse({}) + return None -@item_router.get("/item/{item_uid}/revision/") +@item_router.get("/item/{item_uid}/revision/", response_model=CollectionItemRevisionListResponse) def item_revisions( item_uid: str, limit: int = 50, @@ -468,15 +463,14 @@ def item_revisions( ret_data = [CollectionItemRevisionInOut.from_orm_context(revision, context) for revision in result] iterator = ret_data[-1].uid if len(result) > 0 else None - ret = CollectionItemRevisionListResponse( + return CollectionItemRevisionListResponse( data=ret_data, iterator=iterator, done=done, ) - return MsgpackResponse(ret) -@item_router.post("/item/fetch_updates/") +@item_router.post("/item/fetch_updates/", response_model=CollectionItemListResponse) def fetch_updates( data: t.List[CollectionItemBulkGetIn], stoken: t.Optional[str] = None, @@ -502,12 +496,11 @@ def fetch_updates( new_stoken = new_stoken or stoken context = Context(user, prefetch) - ret = CollectionItemListResponse( + return CollectionItemListResponse( data=[CollectionItemOut.from_orm_context(item, context) for item in queryset], stoken=new_stoken, done=True, # we always return all the items, so it's always done ) - return MsgpackResponse(ret) @item_router.post("/item/transaction/", dependencies=[Depends(has_write_access)]) diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 9b166ee..5c2c338 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -4,14 +4,13 @@ from django.contrib.auth import get_user_model from django.db import transaction, IntegrityError from django.db.models import QuerySet from fastapi import APIRouter, Depends, status, Request -from pydantic import BaseModel from django_etebase import models from django_etebase.utils import get_user_queryset, CallbackContext from .authentication import get_authenticated_user from .exceptions import ValidationError, PermissionDenied -from .msgpack import MsgpackRoute, MsgpackResponse -from .utils import get_object_or_404, Context, is_collection_admin +from .msgpack import MsgpackRoute +from .utils import get_object_or_404, Context, is_collection_admin, BaseModel User = get_user_model() invitation_incoming_router = APIRouter(route_class=MsgpackRoute, tags=["incoming invitation"]) @@ -85,7 +84,7 @@ def list_common( queryset: QuerySet, iterator: t.Optional[str], limit: int, -) -> MsgpackResponse: +) -> InvitationListResponse: queryset = queryset.order_by("id") if iterator is not None: @@ -102,12 +101,11 @@ def list_common( ret_data = result iterator = ret_data[-1].uid if len(result) > 0 else None - ret = InvitationListResponse( + return InvitationListResponse( data=ret_data, iterator=iterator, done=done, ) - return MsgpackResponse(ret) @invitation_incoming_router.get("/", response_model=InvitationListResponse) @@ -125,8 +123,7 @@ def incoming_get( queryset: QuerySet = Depends(get_incoming_queryset), ): obj = get_object_or_404(queryset, uid=invitation_uid) - ret = CollectionInvitationOut.from_orm(obj) - return MsgpackResponse(ret) + return CollectionInvitationOut.from_orm(obj) @invitation_incoming_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) @@ -191,8 +188,6 @@ def outgoing_create( except IntegrityError: raise ValidationError("invitation_exists", "Invitation already exists") - return MsgpackResponse(CollectionInvitationOut.from_orm(ret), status_code=status.HTTP_201_CREATED) - @invitation_outgoing_router.get("/", response_model=InvitationListResponse) def outgoing_list( @@ -221,5 +216,4 @@ def outgoing_fetch_user_profile( kwargs = {User.USERNAME_FIELD: username.lower()} user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) user_info = get_object_or_404(models.UserInfo.objects.all(), owner=user) - ret = UserInfoOut.from_orm(user_info) - return MsgpackResponse(ret) + return UserInfoOut.from_orm(user_info) diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 2c9b631..749092c 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -4,12 +4,11 @@ from django.contrib.auth import get_user_model from django.db import transaction from django.db.models import QuerySet from fastapi import APIRouter, Depends, status -from pydantic import BaseModel from django_etebase import models from .authentication import get_authenticated_user -from .msgpack import MsgpackRoute, MsgpackResponse -from .utils import get_object_or_404 +from .msgpack import MsgpackRoute +from .utils import get_object_or_404, BaseModel from .stoken_handler import filter_by_stoken_and_limit from .collection import get_collection, verify_collection_admin @@ -61,12 +60,11 @@ def member_list( ) new_stoken = new_stoken_obj and new_stoken_obj.uid - ret = MemberListResponse( + return MemberListResponse( data=[CollectionMemberOut.from_orm(item) for item in result], iterator=new_stoken, done=done, ) - return MsgpackResponse(ret) @member_router.delete( diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index 0c5cc30..edffd7e 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -24,7 +24,7 @@ class MsgpackResponse(Response): return b"" if isinstance(content, BaseModel): - content = content.dict(exclude_unset=True) + content = content.dict() return msgpack.packb(content, use_bin_type=True) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 150afe8..7168f87 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -2,6 +2,7 @@ import dataclasses import typing as t from fastapi import status, Query +from pydantic import BaseModel as PyBaseModel from django.db.models import QuerySet from django.core.exceptions import ObjectDoesNotExist @@ -17,6 +18,13 @@ Prefetch = t.Literal["auto", "medium"] PrefetchQuery = Query(default="auto") +class BaseModel(PyBaseModel): + class Config: + json_encoders = { + bytes: lambda x: x, + } + + @dataclasses.dataclass class Context: user: t.Optional[User] From 6517fc5db2dac325028bdb94f937333e9fd42b25 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 09:35:27 +0200 Subject: [PATCH 410/511] More route tags to a central place. --- etebase_fastapi/app.py | 12 ++++++------ etebase_fastapi/authentication.py | 2 +- etebase_fastapi/collection.py | 4 ++-- etebase_fastapi/invitation.py | 4 ++-- etebase_fastapi/member.py | 2 +- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index a8b12d0..2bbfc2a 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -22,12 +22,12 @@ app = FastAPI() VERSION = "v1" BASE_PATH = f"/api/{VERSION}" COLLECTION_UID_MARKER = "{collection_uid}" -app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication") -app.include_router(collection_router, prefix=f"{BASE_PATH}/collection") -app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}") -app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}") -app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming") -app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing") +app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication", tags=["authentication"]) +app.include_router(collection_router, prefix=f"{BASE_PATH}/collection", tags=["collection"]) +app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["item"]) +app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["member"]) +app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"]) +app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"]) if settings.DEBUG: from .test_reset_view import test_reset_view_router diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 13a8884..742f101 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -33,7 +33,7 @@ User = get_user_model() token_scheme = APIKeyHeader(name="Authorization") AUTO_REFRESH = True MIN_REFRESH_INTERVAL = 60 -authentication_router = APIRouter(route_class=MsgpackRoute, tags=["authentication"]) +authentication_router = APIRouter(route_class=MsgpackRoute) @dataclasses.dataclass(frozen=True) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 993d144..8b69708 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -17,8 +17,8 @@ from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_st from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin, BaseModel User = get_user_model() -collection_router = APIRouter(route_class=MsgpackRoute, tags=["collection"]) -item_router = APIRouter(route_class=MsgpackRoute, tags=["item"]) +collection_router = APIRouter(route_class=MsgpackRoute) +item_router = APIRouter(route_class=MsgpackRoute) default_queryset: QuerySet = models.Collection.objects.all() default_item_queryset: QuerySet = models.CollectionItem.objects.all() diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 5c2c338..ab0bf01 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -13,8 +13,8 @@ from .msgpack import MsgpackRoute from .utils import get_object_or_404, Context, is_collection_admin, BaseModel User = get_user_model() -invitation_incoming_router = APIRouter(route_class=MsgpackRoute, tags=["incoming invitation"]) -invitation_outgoing_router = APIRouter(route_class=MsgpackRoute, tags=["outgoing invitation"]) +invitation_incoming_router = APIRouter(route_class=MsgpackRoute) +invitation_outgoing_router = APIRouter(route_class=MsgpackRoute) default_queryset: QuerySet = models.CollectionInvitation.objects.all() diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 749092c..26cfcff 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -14,7 +14,7 @@ from .stoken_handler import filter_by_stoken_and_limit from .collection import get_collection, verify_collection_admin User = get_user_model() -member_router = APIRouter(route_class=MsgpackRoute, tags=["member"]) +member_router = APIRouter(route_class=MsgpackRoute) default_queryset: QuerySet = models.CollectionMember.objects.all() From 34c548acda625c4dbfad57e4d6cd780d4d500250 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 09:47:37 +0200 Subject: [PATCH 411/511] Remove extra import. --- etebase_fastapi/exceptions.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index fa76c45..b7bb0e9 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -59,9 +59,6 @@ class PermissionDenied(CustomHttpException): super().__init__(code=code, detail=detail, status_code=status_code) -from django_etebase.exceptions import EtebaseValidationError - - class ValidationError(CustomHttpException): def __init__( self, From a75d5479faef37c8d046ae0d5ebebb662d9d3645 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 09:51:27 +0200 Subject: [PATCH 412/511] Rename ValidationError to HttpError. --- etebase_fastapi/authentication.py | 16 ++++++++-------- etebase_fastapi/collection.py | 16 ++++++++-------- etebase_fastapi/exceptions.py | 21 +++++++++------------ etebase_fastapi/invitation.py | 6 +++--- etebase_fastapi/utils.py | 4 ++-- 5 files changed, 30 insertions(+), 33 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 742f101..5650652 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -25,7 +25,7 @@ from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user, get_user_queryset, CallbackContext from django_etebase.views import msgpack_encode, msgpack_decode -from .exceptions import AuthenticationFailed, transform_validation_error, ValidationError +from .exceptions import AuthenticationFailed, transform_validation_error, HttpError from .msgpack import MsgpackRoute from .utils import BaseModel @@ -207,20 +207,20 @@ def validate_login_request( challenge_data = msgpack_decode(box.decrypt(validated_data.challenge)) now = int(datetime.now().timestamp()) if validated_data.action != expected_action: - raise ValidationError("wrong_action", f'Expected "{challenge_sent_to_user.response}" but got something else') + raise HttpError("wrong_action", f'Expected "{challenge_sent_to_user.response}" but got something else') elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: - raise ValidationError("challenge_expired", "Login challenge has expired") + raise HttpError("challenge_expired", "Login challenge has expired") elif challenge_data["userId"] != user.id: - raise ValidationError("wrong_user", "This challenge is for the wrong user") + raise HttpError("wrong_user", "This challenge is for the wrong user") elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request: - raise ValidationError( + raise HttpError( "wrong_host", f'Found wrong host name. Got: "{validated_data.host}" expected: "{host_from_request}"' ) verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) try: verify_key.verify(challenge_sent_to_user.response, challenge_sent_to_user.signature) except nacl.exceptions.BadSignatureError: - raise ValidationError("login_bad_signature", "Wrong password for user.", status.HTTP_401_UNAUTHORIZED) + raise HttpError("login_bad_signature", "Wrong password for user.", status.HTTP_401_UNAUTHORIZED) @authentication_router.get("/is_etebase/") @@ -269,7 +269,7 @@ def dashboard_url(user: User = Depends(get_authenticated_user)): # XXX-TOM get_dashboard_url = app_settings.DASHBOARD_URL_FUNC if get_dashboard_url is None: - raise ValidationError("not_supported", "This server doesn't have a user dashboard.") + raise HttpError("not_supported", "This server doesn't have a user dashboard.") ret = { "url": get_dashboard_url(request, *args, **kwargs), @@ -301,7 +301,7 @@ def signup_save(data: SignupIn, request: Request) -> User: raise EtebaseValidationError("generic", str(e)) if hasattr(instance, "userinfo"): - raise ValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) + raise HttpError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) models.UserInfo.objects.create(**data.dict(exclude={"user"}), owner=instance) return instance diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 8b69708..7f01682 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -11,7 +11,7 @@ from fastapi import APIRouter, Depends, status from django_etebase import models from .authentication import get_authenticated_user -from .exceptions import ValidationError, transform_validation_error, PermissionDenied +from .exceptions import HttpError, transform_validation_error, PermissionDenied from .msgpack import MsgpackRoute from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin, BaseModel @@ -144,7 +144,7 @@ class ItemDepIn(BaseModel): item = models.CollectionItem.objects.get(uid=self.uid) etag = self.etag if item.etag != etag: - raise ValidationError( + raise HttpError( "wrong_etag", "Wrong etag. Expected {} got {}".format(item.etag, etag), status_code=status.HTTP_409_CONFLICT, @@ -274,7 +274,7 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) chunk_obj.save() else: - raise ValidationError("chunk_no_content", "Tried to create a new chunk without content") + raise HttpError("chunk_no_content", "Tried to create a new chunk without content") chunks_objs.append(chunk_obj) @@ -290,12 +290,12 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle def _create(data: CollectionIn, user: User): with transaction.atomic(): if data.item.etag is not None: - raise ValidationError("bad_etag", "etag is not null") + raise HttpError("bad_etag", "etag is not null") instance = models.Collection(uid=data.item.uid, owner=user) try: instance.validate_unique() except django_exceptions.ValidationError: - raise ValidationError( + raise HttpError( "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT ) instance.save() @@ -355,7 +355,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance if validate_etag and cur_etag != etag: - raise ValidationError( + raise HttpError( "wrong_etag", "Wrong etag. Expected {} got {}".format(cur_etag, etag), status_code=status.HTTP_409_CONFLICT, @@ -425,7 +425,7 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid collection_object = queryset.select_for_update().get(uid=uid) if stoken is not None and stoken != collection_object.stoken: - raise ValidationError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) + raise HttpError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) # XXX-TOM: make sure we return compatible errors data.validate_db() @@ -482,7 +482,7 @@ def fetch_updates( item_limit = 200 if len(data) > item_limit: - raise ValidationError("too_many_items", "Request has too many items.", status_code=status.HTTP_400_BAD_REQUEST) + raise HttpError("too_many_items", "Request has too many items.", status_code=status.HTTP_400_BAD_REQUEST) queryset, stoken_rev = filter_by_stoken(stoken, queryset, models.CollectionItem.stoken_annotation) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index b7bb0e9..2c1757c 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -3,19 +3,17 @@ import typing as t from pydantic import BaseModel -from django_etebase.exceptions import EtebaseValidationError - -class ValidationErrorField(BaseModel): +class HttpErrorField(BaseModel): field: str code: str detail: str -class ValidationErrorOut(BaseModel): +class HttpErrorOut(BaseModel): code: str detail: str - errors: t.Optional[t.List[ValidationErrorField]] + errors: t.Optional[t.List[HttpErrorField]] class CustomHttpException(Exception): @@ -59,24 +57,23 @@ class PermissionDenied(CustomHttpException): super().__init__(code=code, detail=detail, status_code=status_code) -class ValidationError(CustomHttpException): +class HttpError(CustomHttpException): def __init__( self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST, - field: t.Optional[str] = None, - errors: t.Optional[t.List["ValidationError"]] = None, + errors: t.Optional[t.List["HttpError"]] = None, ): self.errors = errors super().__init__(code=code, detail=detail, status_code=status_code) @property def as_dict(self) -> dict: - return ValidationErrorOut(code=self.code, errors=self.errors, detail=self.detail).dict() + return HttpErrorOut(code=self.code, errors=self.errors, detail=self.detail).dict() -def flatten_errors(field_name, errors) -> t.List[ValidationError]: +def flatten_errors(field_name, errors) -> t.List[HttpError]: ret = [] if isinstance(errors, dict): for error_key in errors: @@ -98,5 +95,5 @@ def transform_validation_error(prefix, err): elif not hasattr(err, "message"): errors = flatten_errors(prefix, err.error_list) else: - raise EtebaseValidationError(err.code, err.message) - raise ValidationError(code="field_errors", detail="Field validations failed.", errors=errors) + raise HttpError(err.code, err.message) + raise HttpError(code="field_errors", detail="Field validations failed.", errors=errors) diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index ab0bf01..38b74d8 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -8,7 +8,7 @@ from fastapi import APIRouter, Depends, status, Request from django_etebase import models from django_etebase.utils import get_user_queryset, CallbackContext from .authentication import get_authenticated_user -from .exceptions import ValidationError, PermissionDenied +from .exceptions import HttpError, PermissionDenied from .msgpack import MsgpackRoute from .utils import get_object_or_404, Context, is_collection_admin, BaseModel @@ -42,7 +42,7 @@ class CollectionInvitationCommon(BaseModel): class CollectionInvitationIn(CollectionInvitationCommon): def validate_db(self, context: Context): if context.user.username == self.username.lower(): - raise ValidationError("no_self_invite", "Inviting yourself is not allowed") + raise HttpError("no_self_invite", "Inviting yourself is not allowed") class CollectionInvitationOut(CollectionInvitationCommon): @@ -186,7 +186,7 @@ def outgoing_create( **data.dict(exclude={"collection", "username"}), user=to_user, fromMember=member ) except IntegrityError: - raise ValidationError("invitation_exists", "Invitation already exists") + raise HttpError("invitation_exists", "Invitation already exists") @invitation_outgoing_router.get("/", response_model=InvitationListResponse) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 7168f87..6ea9513 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -10,7 +10,7 @@ from django.contrib.auth import get_user_model from django_etebase.models import AccessLevels -from .exceptions import ValidationError +from .exceptions import HttpError User = get_user_model() @@ -35,7 +35,7 @@ def get_object_or_404(queryset: QuerySet, **kwargs): try: return queryset.get(**kwargs) except ObjectDoesNotExist as e: - raise ValidationError("does_not_exist", str(e), status_code=status.HTTP_404_NOT_FOUND) + raise HttpError("does_not_exist", str(e), status_code=status.HTTP_404_NOT_FOUND) def is_collection_admin(collection, user): From 4b4be14d32330a6df16b51eb2c93d484402d157a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:00:35 +0200 Subject: [PATCH 413/511] Add more responses to the API. --- etebase_fastapi/authentication.py | 10 +++++----- etebase_fastapi/collection.py | 6 +++--- etebase_fastapi/invitation.py | 6 +++--- etebase_fastapi/member.py | 4 ++-- etebase_fastapi/utils.py | 6 +++++- 5 files changed, 18 insertions(+), 14 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 5650652..902b79b 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -14,7 +14,7 @@ from django.contrib.auth import get_user_model, user_logged_out, user_logged_in from django.core import exceptions as django_exceptions from django.db import transaction from django.utils import timezone -from fastapi import APIRouter, Depends, status, Request, Response +from fastapi import APIRouter, Depends, status, Request from fastapi.security import APIKeyHeader from django_etebase import app_settings, models @@ -27,7 +27,7 @@ from django_etebase.utils import create_user, get_user_queryset, CallbackContext from django_etebase.views import msgpack_encode, msgpack_decode from .exceptions import AuthenticationFailed, transform_validation_error, HttpError from .msgpack import MsgpackRoute -from .utils import BaseModel +from .utils import BaseModel, permission_responses User = get_user_model() token_scheme = APIKeyHeader(name="Authorization") @@ -250,21 +250,21 @@ async def login(data: Login, request: Request): return data -@authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT) +@authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) async def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): await sync_to_async(auth_data.token.delete)() # XXX-TOM await sync_to_async(user_logged_out.send)(sender=auth_data.user.__class__, request=None, user=auth_data.user) -@authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT) +@authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) async def change_password(data: ChangePassword, request: Request, user: User = Depends(get_authenticated_user)): host = request.headers.get("Host") await validate_login_request(data.response_data, data, user, "changePassword", host) await sync_to_async(save_changed_password)(data, user) -@authentication_router.post("/dashboard_url/") +@authentication_router.post("/dashboard_url/", responses=permission_responses) def dashboard_url(user: User = Depends(get_authenticated_user)): # XXX-TOM get_dashboard_url = app_settings.DASHBOARD_URL_FUNC diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 7f01682..fad49aa 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -14,11 +14,11 @@ from .authentication import get_authenticated_user from .exceptions import HttpError, transform_validation_error, PermissionDenied from .msgpack import MsgpackRoute from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken -from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin, BaseModel +from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin, BaseModel, permission_responses User = get_user_model() -collection_router = APIRouter(route_class=MsgpackRoute) -item_router = APIRouter(route_class=MsgpackRoute) +collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) default_queryset: QuerySet = models.Collection.objects.all() default_item_queryset: QuerySet = models.CollectionItem.objects.all() diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 38b74d8..1d8df94 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -10,11 +10,11 @@ from django_etebase.utils import get_user_queryset, CallbackContext from .authentication import get_authenticated_user from .exceptions import HttpError, PermissionDenied from .msgpack import MsgpackRoute -from .utils import get_object_or_404, Context, is_collection_admin, BaseModel +from .utils import get_object_or_404, Context, is_collection_admin, BaseModel, permission_responses User = get_user_model() -invitation_incoming_router = APIRouter(route_class=MsgpackRoute) -invitation_outgoing_router = APIRouter(route_class=MsgpackRoute) +invitation_incoming_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +invitation_outgoing_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) default_queryset: QuerySet = models.CollectionInvitation.objects.all() diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 26cfcff..8ffed9d 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -8,13 +8,13 @@ from fastapi import APIRouter, Depends, status from django_etebase import models from .authentication import get_authenticated_user from .msgpack import MsgpackRoute -from .utils import get_object_or_404, BaseModel +from .utils import get_object_or_404, BaseModel, permission_responses from .stoken_handler import filter_by_stoken_and_limit from .collection import get_collection, verify_collection_admin User = get_user_model() -member_router = APIRouter(route_class=MsgpackRoute) +member_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) default_queryset: QuerySet = models.CollectionMember.objects.all() diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 6ea9513..487f03a 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -10,7 +10,7 @@ from django.contrib.auth import get_user_model from django_etebase.models import AccessLevels -from .exceptions import HttpError +from .exceptions import HttpError, HttpErrorOut User = get_user_model() @@ -41,3 +41,7 @@ def get_object_or_404(queryset: QuerySet, **kwargs): def is_collection_admin(collection, user): member = collection.members.filter(user=user).first() return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) + + +response_model_dict = {"model": HttpErrorOut} +permission_responses = {403: response_model_dict} From b39f7951e292588092365964f7c2fc3f317c54d3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:18:35 +0200 Subject: [PATCH 414/511] chunk first-type. --- etebase_fastapi/collection.py | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index fad49aa..20d79e0 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -515,3 +515,31 @@ def item_batch( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) + + +# Chunks + + +@item_router.put("/item/{item_uid}/chunk/{chunk_uid}/", dependencies=[Depends(has_write_access)], status_code=status.HTTP_201_CREATED) +def chunk_update( + limit: int = 50, + iterator: t.Optional[str] = None, + prefetch: Prefetch = PrefetchQuery, + user: User = Depends(get_authenticated_user), + collection: models.Collection = Depends(get_collection), +): + # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) + + data = { + "uid": chunk_uid, + "chunkFile": request.data["file"], + } + + serializer = self.get_serializer_class()(data=data) + serializer.is_valid(raise_exception=True) + try: + serializer.save(collection=col) + except IntegrityError: + return Response( + {"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT + ) From 959dc9b576ddeeb69de6f367fd381d217a1bccb7 Mon Sep 17 00:00:00 2001 From: Tal Leibman Date: Mon, 28 Dec 2020 10:27:49 +0200 Subject: [PATCH 415/511] minor fix --- etebase_fastapi/collection.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 20d79e0..6757ac3 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -70,7 +70,6 @@ class CollectionItemOut(CollectionItemCommon): uid=obj.uid, version=obj.version, encryptionKey=obj.encryptionKey, - etag=obj.etag, content=CollectionItemRevisionInOut.from_orm_context(obj.content, context), ) @@ -91,7 +90,7 @@ class CollectionOut(CollectionCommon): @classmethod def from_orm_context(cls: t.Type["CollectionOut"], obj: models.Collection, context: Context) -> "CollectionOut": - member: CollectionMember = obj.members.get(user=context.user) + member: models.CollectionMember = obj.members.get(user=context.user) collection_type = member.collectionType ret = cls( collectionType=collection_type and collection_type.uid, From 1a09393dcb425250fc92c73b3584884f4f96e6c8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:29:47 +0200 Subject: [PATCH 416/511] Also add 401 to permission responses. --- etebase_fastapi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 487f03a..2ee3700 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -44,4 +44,4 @@ def is_collection_admin(collection, user): response_model_dict = {"model": HttpErrorOut} -permission_responses = {403: response_model_dict} +permission_responses = {401: response_model_dict, 403: response_model_dict} From 37f5a4509f700169cdd60e5f47ccc69034fc9302 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:41:22 +0200 Subject: [PATCH 417/511] Improve chunks type. --- etebase_fastapi/collection.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 6757ac3..3af5af0 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -31,7 +31,10 @@ class CollectionItemRevisionInOut(BaseModel): uid: str meta: bytes deleted: bool - chunks: t.List[t.Tuple[str, t.Optional[bytes]]] + chunks: t.List[t.Union[ + t.Tuple[str], + t.Tuple[str, bytes], + ]] class Config: orm_mode = True From cf7690a60f23bbaee363ad15f0415384bf6f037e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:45:34 +0200 Subject: [PATCH 418/511] Remove usages of EtebaseValidationError. --- etebase_fastapi/authentication.py | 2 +- etebase_fastapi/stoken_handler.py | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 902b79b..a13cc51 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -298,7 +298,7 @@ def signup_save(data: SignupIn, request: Request) -> User: except django_exceptions.ValidationError as e: transform_validation_error("user", e) except Exception as e: - raise EtebaseValidationError("generic", str(e)) + raise HttpError("generic", str(e)) if hasattr(instance, "userinfo"): raise HttpError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py index a976830..76d348a 100644 --- a/etebase_fastapi/stoken_handler.py +++ b/etebase_fastapi/stoken_handler.py @@ -3,9 +3,10 @@ import typing as t from django.db.models import QuerySet from fastapi import status -from django_etebase.exceptions import EtebaseValidationError from django_etebase.models import Stoken +from .exceptions import HttpError + # TODO missing stoken_annotation type StokenAnnotation = t.Any @@ -15,7 +16,7 @@ def get_stoken_obj(stoken: t.Optional[str]) -> t.Optional[Stoken]: try: return Stoken.objects.get(uid=stoken) except Stoken.DoesNotExist: - raise EtebaseValidationError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) + raise HttpError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) return None From 38884fead8cf91783e0213a948459c98973bb7db Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:47:07 +0200 Subject: [PATCH 419/511] Revert "Improve chunks type." This reverts commit 37f5a4509f700169cdd60e5f47ccc69034fc9302. --- etebase_fastapi/collection.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 3af5af0..6757ac3 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -31,10 +31,7 @@ class CollectionItemRevisionInOut(BaseModel): uid: str meta: bytes deleted: bool - chunks: t.List[t.Union[ - t.Tuple[str], - t.Tuple[str, bytes], - ]] + chunks: t.List[t.Tuple[str, t.Optional[bytes]]] class Config: orm_mode = True From ad2205e59616c90256afbb908ed95afaa2c24482 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 10:57:40 +0200 Subject: [PATCH 420/511] Add trusted host middleware. --- etebase_fastapi/app.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 2bbfc2a..75cb099 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -1,7 +1,6 @@ import os from django.core.wsgi import get_wsgi_application -from fastapi.middleware.cors import CORSMiddleware os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") application = get_wsgi_application() @@ -10,6 +9,8 @@ from django.conf import settings # Not at the top of the file because we first need to setup django from fastapi import FastAPI, Request +from fastapi.middleware.cors import CORSMiddleware +from fastapi.middleware.trustedhost import TrustedHostMiddleware from .exceptions import CustomHttpException from .authentication import authentication_router @@ -35,6 +36,7 @@ if settings.DEBUG: app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] ) +app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) @app.exception_handler(CustomHttpException) From 08821c5e332faef831a4ba18cf6dc5645da4d583 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 16:39:16 +0200 Subject: [PATCH 421/511] Update changelog. --- ChangeLog.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index e3c8232..a74a8af 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,13 @@ # Changelog +## Version 0.7.0 +* Chunks: improve the chunk download endpoint to use sendfile extensions +* Chunks: support not passing chunk content if exists +* Chunks: fix chunk uploading media type to accept everything +* Gracefull handle uploading the same revision +* Pass generic context to callbacks instead of the whole view +* Fix handling of some validation errors + ## Version 0.6.1 * Collection: save the UID on the model to use the db for enforcing uniqueness From 295ae6f3d34b6a6732a7789081a59c45a7de4733 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 16:39:16 +0200 Subject: [PATCH 422/511] Update changelog. --- ChangeLog.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index e3c8232..a74a8af 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,13 @@ # Changelog +## Version 0.7.0 +* Chunks: improve the chunk download endpoint to use sendfile extensions +* Chunks: support not passing chunk content if exists +* Chunks: fix chunk uploading media type to accept everything +* Gracefull handle uploading the same revision +* Pass generic context to callbacks instead of the whole view +* Fix handling of some validation errors + ## Version 0.6.1 * Collection: save the UID on the model to use the db for enforcing uniqueness From 63afcc0830170834e59b4db17301743d3c636749 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 11:44:17 +0200 Subject: [PATCH 423/511] Mount the django application. --- etebase_fastapi/app.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/etebase_fastapi/app.py b/etebase_fastapi/app.py index 75cb099..6dffef4 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/app.py @@ -9,6 +9,7 @@ from django.conf import settings # Not at the top of the file because we first need to setup django from fastapi import FastAPI, Request +from fastapi.middleware.wsgi import WSGIMiddleware from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware @@ -37,6 +38,7 @@ app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] ) app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) +app.mount("/", WSGIMiddleware(application)) @app.exception_handler(CustomHttpException) From 6c05a7898a14bd81f42ac0fd7c6760785ecb1d78 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 11:49:20 +0200 Subject: [PATCH 424/511] Add functions to split read and write permissions. --- django_etebase/app_settings.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index 7c93f5f..c1e8dc9 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -33,14 +33,36 @@ class AppSettings: @cached_property def API_PERMISSIONS(self): # pylint: disable=invalid-name + """ + Deprecated. Do not use. + """ perms = self._setting("API_PERMISSIONS", ("rest_framework.permissions.IsAuthenticated",)) ret = [] for perm in perms: ret.append(self.import_from_str(perm)) return ret + @cached_property + def API_PERMISSIONS_READ(self): # pylint: disable=invalid-name + perms = self._setting("API_PERMISSIONS_READ", tuple()) + ret = [] + for perm in perms: + ret.append(self.import_from_str(perm)) + return ret + + @cached_property + def API_PERMISSIONS_WRITE(self): # pylint: disable=invalid-name + perms = self._setting("API_PERMISSIONS_WRITE", tuple()) + ret = [] + for perm in perms: + ret.append(self.import_from_str(perm)) + return ret + @cached_property def API_AUTHENTICATORS(self): # pylint: disable=invalid-name + """ + Deprecated. Do not use. + """ perms = self._setting( "API_AUTHENTICATORS", ( From b081d0129fafb02434fe94be69e3216f3ad74ac7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 12:12:00 +0200 Subject: [PATCH 425/511] Add support for read/write permissions. --- etebase_fastapi/collection.py | 61 +++++++++++++++++++++++------------ etebase_fastapi/invitation.py | 34 +++++++++++++------ etebase_fastapi/member.py | 15 ++++----- etebase_fastapi/utils.py | 7 +++- etebase_server/settings.py | 5 --- 5 files changed, 79 insertions(+), 43 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 6757ac3..e6c10c3 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -14,7 +14,17 @@ from .authentication import get_authenticated_user from .exceptions import HttpError, transform_validation_error, PermissionDenied from .msgpack import MsgpackRoute from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken -from .utils import get_object_or_404, Context, Prefetch, PrefetchQuery, is_collection_admin, BaseModel, permission_responses +from .utils import ( + get_object_or_404, + Context, + Prefetch, + PrefetchQuery, + is_collection_admin, + BaseModel, + permission_responses, + PERMISSIONS_READ, + PERMISSIONS_READWRITE, +) User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -228,7 +238,13 @@ def has_write_access( # paths -@collection_router.post("/list_multi/", response_model=CollectionListResponse, response_model_exclude_unset=True) + +@collection_router.post( + "/list_multi/", + response_model=CollectionListResponse, + response_model_exclude_unset=True, + dependencies=PERMISSIONS_READ, +) async def list_multi( data: ListMulti, stoken: t.Optional[str] = None, @@ -245,7 +261,7 @@ async def list_multi( return await collection_list_common(queryset, user, stoken, limit, prefetch) -@collection_router.get("/", response_model=CollectionListResponse) +@collection_router.get("/", response_model=CollectionListResponse, dependencies=PERMISSIONS_READ) async def collection_list( stoken: t.Optional[str] = None, limit: int = 50, @@ -321,17 +337,17 @@ def _create(data: CollectionIn, user: User): ).save() -@collection_router.post("/", status_code=status.HTTP_201_CREATED) +@collection_router.post("/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE) async def create(data: CollectionIn, user: User = Depends(get_authenticated_user)): await sync_to_async(_create)(data, user) -@collection_router.get("/{collection_uid}/", response_model=CollectionOut) +@collection_router.get("/{collection_uid}/", response_model=CollectionOut, dependencies=PERMISSIONS_READ) def collection_get( - obj: models.Collection = Depends(get_collection), - user: User = Depends(get_authenticated_user), - prefetch: Prefetch = PrefetchQuery - ): + obj: models.Collection = Depends(get_collection), + user: User = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery, +): return CollectionOut.from_orm_context(obj, Context(user, prefetch)) @@ -375,11 +391,12 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance -@item_router.get("/item/{item_uid}/", response_model=CollectionItemOut) +@item_router.get("/item/{item_uid}/", response_model=CollectionItemOut, dependencies=PERMISSIONS_READ) def item_get( item_uid: str, queryset: QuerySet = Depends(get_item_queryset), - user: User = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, + user: User = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery, ): obj = queryset.get(uid=item_uid) return CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) @@ -402,7 +419,7 @@ def item_list_common( return CollectionItemListResponse(data=data, stoken=new_stoken, done=done) -@item_router.get("/item/", response_model=CollectionItemListResponse) +@item_router.get("/item/", response_model=CollectionItemListResponse, dependencies=PERMISSIONS_READ) async def item_list( queryset: QuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, @@ -434,7 +451,9 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid return None -@item_router.get("/item/{item_uid}/revision/", response_model=CollectionItemRevisionListResponse) +@item_router.get( + "/item/{item_uid}/revision/", response_model=CollectionItemRevisionListResponse, dependencies=PERMISSIONS_READ +) def item_revisions( item_uid: str, limit: int = 50, @@ -469,7 +488,7 @@ def item_revisions( ) -@item_router.post("/item/fetch_updates/", response_model=CollectionItemListResponse) +@item_router.post("/item/fetch_updates/", response_model=CollectionItemListResponse, dependencies=PERMISSIONS_READ) def fetch_updates( data: t.List[CollectionItemBulkGetIn], stoken: t.Optional[str] = None, @@ -502,14 +521,14 @@ def fetch_updates( ) -@item_router.post("/item/transaction/", dependencies=[Depends(has_write_access)]) +@item_router.post("/item/transaction/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_transaction( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) -@item_router.post("/item/batch/", dependencies=[Depends(has_write_access)]) +@item_router.post("/item/batch/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_batch( collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) ): @@ -519,7 +538,11 @@ def item_batch( # Chunks -@item_router.put("/item/{item_uid}/chunk/{chunk_uid}/", dependencies=[Depends(has_write_access)], status_code=status.HTTP_201_CREATED) +@item_router.put( + "/item/{item_uid}/chunk/{chunk_uid}/", + dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE], + status_code=status.HTTP_201_CREATED, +) def chunk_update( limit: int = 50, iterator: t.Optional[str] = None, @@ -539,6 +562,4 @@ def chunk_update( try: serializer.save(collection=col) except IntegrityError: - return Response( - {"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT - ) + return Response({"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT) diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 1d8df94..39460a9 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -10,7 +10,15 @@ from django_etebase.utils import get_user_queryset, CallbackContext from .authentication import get_authenticated_user from .exceptions import HttpError, PermissionDenied from .msgpack import MsgpackRoute -from .utils import get_object_or_404, Context, is_collection_admin, BaseModel, permission_responses +from .utils import ( + get_object_or_404, + Context, + is_collection_admin, + BaseModel, + permission_responses, + PERMISSIONS_READ, + PERMISSIONS_READWRITE, +) User = get_user_model() invitation_incoming_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -108,7 +116,7 @@ def list_common( ) -@invitation_incoming_router.get("/", response_model=InvitationListResponse) +@invitation_incoming_router.get("/", response_model=InvitationListResponse, dependencies=PERMISSIONS_READ) def incoming_list( iterator: t.Optional[str] = None, limit: int = 50, @@ -117,7 +125,9 @@ def incoming_list( return list_common(queryset, iterator, limit) -@invitation_incoming_router.get("/{invitation_uid}/", response_model=CollectionInvitationOut) +@invitation_incoming_router.get( + "/{invitation_uid}/", response_model=CollectionInvitationOut, dependencies=PERMISSIONS_READ +) def incoming_get( invitation_uid: str, queryset: QuerySet = Depends(get_incoming_queryset), @@ -126,7 +136,9 @@ def incoming_get( return CollectionInvitationOut.from_orm(obj) -@invitation_incoming_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) +@invitation_incoming_router.delete( + "/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READWRITE +) def incoming_delete( invitation_uid: str, queryset: QuerySet = Depends(get_incoming_queryset), @@ -135,7 +147,9 @@ def incoming_delete( obj.delete() -@invitation_incoming_router.post("/{invitation_uid}/accept/", status_code=status.HTTP_201_CREATED) +@invitation_incoming_router.post( + "/{invitation_uid}/accept/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE +) def incoming_accept( invitation_uid: str, data: CollectionInvitationAcceptIn, @@ -161,7 +175,7 @@ def incoming_accept( invitation.delete() -@invitation_outgoing_router.post("/", status_code=status.HTTP_201_CREATED) +@invitation_outgoing_router.post("/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE) def outgoing_create( data: CollectionInvitationIn, request: Request, @@ -189,7 +203,7 @@ def outgoing_create( raise HttpError("invitation_exists", "Invitation already exists") -@invitation_outgoing_router.get("/", response_model=InvitationListResponse) +@invitation_outgoing_router.get("/", response_model=InvitationListResponse, dependencies=PERMISSIONS_READ) def outgoing_list( iterator: t.Optional[str] = None, limit: int = 50, @@ -198,7 +212,9 @@ def outgoing_list( return list_common(queryset, iterator, limit) -@invitation_outgoing_router.delete("/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT) +@invitation_outgoing_router.delete( + "/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READWRITE +) def outgoing_delete( invitation_uid: str, queryset: QuerySet = Depends(get_outgoing_queryset), @@ -207,7 +223,7 @@ def outgoing_delete( obj.delete() -@invitation_outgoing_router.get("/fetch_user_profile/", response_model=UserInfoOut) +@invitation_outgoing_router.get("/fetch_user_profile/", response_model=UserInfoOut, dependencies=PERMISSIONS_READ) def outgoing_fetch_user_profile( username: str, request: Request, diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 8ffed9d..725d44b 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -8,7 +8,7 @@ from fastapi import APIRouter, Depends, status from django_etebase import models from .authentication import get_authenticated_user from .msgpack import MsgpackRoute -from .utils import get_object_or_404, BaseModel, permission_responses +from .utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE from .stoken_handler import filter_by_stoken_and_limit from .collection import get_collection, verify_collection_admin @@ -48,7 +48,9 @@ class MemberListResponse(BaseModel): done: bool -@member_router.get("/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin)]) +@member_router.get( + "/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin), *PERMISSIONS_READ] +) def member_list( iterator: t.Optional[str] = None, limit: int = 50, @@ -70,7 +72,7 @@ def member_list( @member_router.delete( "/member/{username}/", status_code=status.HTTP_204_NO_CONTENT, - dependencies=[Depends(verify_collection_admin)], + dependencies=[Depends(verify_collection_admin), *PERMISSIONS_READWRITE], ) def member_delete( obj: models.CollectionMember = Depends(get_member), @@ -81,7 +83,7 @@ def member_delete( @member_router.patch( "/member/{username}/", status_code=status.HTTP_204_NO_CONTENT, - dependencies=[Depends(verify_collection_admin)], + dependencies=[Depends(verify_collection_admin), *PERMISSIONS_READWRITE], ) def member_patch( data: CollectionMemberModifyAccessLevelIn, @@ -95,10 +97,7 @@ def member_patch( instance.save() -@member_router.post( - "/member/leave/", - status_code=status.HTTP_204_NO_CONTENT, -) +@member_router.post("/member/leave/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READ) def member_leave(user: User = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection)): obj = get_object_or_404(collection.members, user=user) obj.revoke() diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 2ee3700..165163a 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -1,13 +1,14 @@ import dataclasses import typing as t -from fastapi import status, Query +from fastapi import status, Query, Depends from pydantic import BaseModel as PyBaseModel from django.db.models import QuerySet from django.core.exceptions import ObjectDoesNotExist from django.contrib.auth import get_user_model +from django_etebase import app_settings from django_etebase.models import AccessLevels from .exceptions import HttpError, HttpErrorOut @@ -43,5 +44,9 @@ def is_collection_admin(collection, user): return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) +PERMISSIONS_READ = [Depends(x) for x in app_settings.API_PERMISSIONS_READ] +PERMISSIONS_READWRITE = PERMISSIONS_READ + [Depends(x) for x in app_settings.API_PERMISSIONS_WRITE] + + response_model_dict = {"model": HttpErrorOut} permission_responses = {401: response_model_dict, 403: response_model_dict} diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 325dca9..46ad3c9 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -166,11 +166,6 @@ if any(os.path.isfile(x) for x in config_locations): if "database" in config: DATABASES = {"default": {x.upper(): y for x, y in config.items("database")}} -ETEBASE_API_PERMISSIONS = ("rest_framework.permissions.IsAuthenticated",) -ETEBASE_API_AUTHENTICATORS = ( - "django_etebase.token_auth.authentication.TokenAuthentication", - "rest_framework.authentication.SessionAuthentication", -) ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked" # Efficient file streaming (for large files) From 0fa2f2da3b0c1064c27b5e2851c147843c75a450 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 13:26:12 +0200 Subject: [PATCH 426/511] Make the fastapi application the main asgi one. --- etebase_fastapi/{app.py => main.py} | 13 +++---------- etebase_server/asgi.py | 21 +++++++++++---------- etebase_server/urls.py | 15 ++++++++++++++- etebase_server/wsgi.py | 16 ---------------- 4 files changed, 28 insertions(+), 37 deletions(-) rename etebase_fastapi/{app.py => main.py} (85%) delete mode 100644 etebase_server/wsgi.py diff --git a/etebase_fastapi/app.py b/etebase_fastapi/main.py similarity index 85% rename from etebase_fastapi/app.py rename to etebase_fastapi/main.py index 6dffef4..2c10854 100644 --- a/etebase_fastapi/app.py +++ b/etebase_fastapi/main.py @@ -1,15 +1,7 @@ -import os - -from django.core.wsgi import get_wsgi_application - -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") -application = get_wsgi_application() - from django.conf import settings # Not at the top of the file because we first need to setup django from fastapi import FastAPI, Request -from fastapi.middleware.wsgi import WSGIMiddleware from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware @@ -30,15 +22,16 @@ app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_ app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["member"]) app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"]) app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"]) + if settings.DEBUG: - from .test_reset_view import test_reset_view_router + from etebase_fastapi.test_reset_view import test_reset_view_router app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") + app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] ) app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) -app.mount("/", WSGIMiddleware(application)) @app.exception_handler(CustomHttpException) diff --git a/etebase_server/asgi.py b/etebase_server/asgi.py index 0bf63ec..92fad1c 100644 --- a/etebase_server/asgi.py +++ b/etebase_server/asgi.py @@ -1,16 +1,17 @@ -""" -ASGI config for etebase_server project. - -It exposes the ASGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/ -""" - import os from django.core.asgi import get_asgi_application os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") +django_application = get_asgi_application() + + +def create_application(): + from etebase_fastapi.main import app + + app.mount("/", django_application) + + return app + -application = get_asgi_application() +application = create_application() diff --git a/etebase_server/urls.py b/etebase_server/urls.py index f285977..443763d 100644 --- a/etebase_server/urls.py +++ b/etebase_server/urls.py @@ -1,8 +1,12 @@ +import os + from django.conf import settings from django.conf.urls import include, url from django.contrib import admin -from django.urls import path +from django.urls import path, re_path from django.views.generic import TemplateView +from django.views.static import serve +from django.contrib.staticfiles import finders urlpatterns = [ url(r"^api/", include("django_etebase.urls")), @@ -14,3 +18,12 @@ if settings.DEBUG: urlpatterns += [ url(r"^api-auth/", include("rest_framework.urls", namespace="rest_framework")), ] + + def serve_static(request, path): + filename = finders.find(path) + dirname = os.path.dirname(filename) + basename = os.path.basename(filename) + + return serve(request, basename, dirname) + + urlpatterns += [re_path(r"^static/(?P.*)$", serve_static)] diff --git a/etebase_server/wsgi.py b/etebase_server/wsgi.py deleted file mode 100644 index 908f88c..0000000 --- a/etebase_server/wsgi.py +++ /dev/null @@ -1,16 +0,0 @@ -""" -WSGI config for etebase_server project. - -It exposes the WSGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/3.0/howto/deployment/wsgi/ -""" - -import os - -from django.core.wsgi import get_wsgi_application - -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") - -application = get_wsgi_application() From 4ceb42780ec3d8c475601e2e0b360fb50ceca21b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 13:56:53 +0200 Subject: [PATCH 427/511] Remove unused django_etebase code. --- README.md | 2 +- django_etebase/admin.py | 3 - django_etebase/drf_msgpack/__init__.py | 0 django_etebase/drf_msgpack/apps.py | 5 - .../drf_msgpack/migrations/__init__.py | 0 django_etebase/drf_msgpack/parsers.py | 14 - django_etebase/drf_msgpack/renderers.py | 15 - django_etebase/drf_msgpack/views.py | 3 - django_etebase/exceptions.py | 12 - django_etebase/parsers.py | 14 - django_etebase/permissions.py | 93 -- django_etebase/renderers.py | 19 - django_etebase/serializers.py | 598 ------------ django_etebase/tests.py | 3 - django_etebase/token_auth/admin.py | 0 django_etebase/token_auth/authentication.py | 46 - django_etebase/urls.py | 30 - django_etebase/views.py | 861 ------------------ etebase_fastapi/authentication.py | 6 +- etebase_fastapi/utils.py | 9 + etebase_server/settings.py | 6 - etebase_server/urls.py | 6 +- requirements.in/base.txt | 6 +- requirements.txt | 6 +- 24 files changed, 15 insertions(+), 1742 deletions(-) delete mode 100644 django_etebase/admin.py delete mode 100644 django_etebase/drf_msgpack/__init__.py delete mode 100644 django_etebase/drf_msgpack/apps.py delete mode 100644 django_etebase/drf_msgpack/migrations/__init__.py delete mode 100644 django_etebase/drf_msgpack/parsers.py delete mode 100644 django_etebase/drf_msgpack/renderers.py delete mode 100644 django_etebase/drf_msgpack/views.py delete mode 100644 django_etebase/exceptions.py delete mode 100644 django_etebase/parsers.py delete mode 100644 django_etebase/permissions.py delete mode 100644 django_etebase/renderers.py delete mode 100644 django_etebase/serializers.py delete mode 100644 django_etebase/tests.py delete mode 100644 django_etebase/token_auth/admin.py delete mode 100644 django_etebase/token_auth/authentication.py delete mode 100644 django_etebase/urls.py delete mode 100644 django_etebase/views.py diff --git a/README.md b/README.md index 3e0bd53..1787a2f 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ Now you can initialise our django app. And you are done! You can now run the debug server just to see everything works as expected by running: ``` -./manage.py runserver 0.0.0.0:8000 +uvicorn etebase_server.asgi:application --port 8000 ``` Using the debug server in production is not recommended, so please read the following section for a proper deployment. diff --git a/django_etebase/admin.py b/django_etebase/admin.py deleted file mode 100644 index 8c38f3f..0000000 --- a/django_etebase/admin.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.contrib import admin - -# Register your models here. diff --git a/django_etebase/drf_msgpack/__init__.py b/django_etebase/drf_msgpack/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/django_etebase/drf_msgpack/apps.py b/django_etebase/drf_msgpack/apps.py deleted file mode 100644 index 22ea2c1..0000000 --- a/django_etebase/drf_msgpack/apps.py +++ /dev/null @@ -1,5 +0,0 @@ -from django.apps import AppConfig - - -class DrfMsgpackConfig(AppConfig): - name = "drf_msgpack" diff --git a/django_etebase/drf_msgpack/migrations/__init__.py b/django_etebase/drf_msgpack/migrations/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/django_etebase/drf_msgpack/parsers.py b/django_etebase/drf_msgpack/parsers.py deleted file mode 100644 index 0504a76..0000000 --- a/django_etebase/drf_msgpack/parsers.py +++ /dev/null @@ -1,14 +0,0 @@ -import msgpack - -from rest_framework.parsers import BaseParser -from rest_framework.exceptions import ParseError - - -class MessagePackParser(BaseParser): - media_type = "application/msgpack" - - def parse(self, stream, media_type=None, parser_context=None): - try: - return msgpack.unpackb(stream.read(), raw=False) - except Exception as exc: - raise ParseError("MessagePack parse error - %s" % str(exc)) diff --git a/django_etebase/drf_msgpack/renderers.py b/django_etebase/drf_msgpack/renderers.py deleted file mode 100644 index 35a4afa..0000000 --- a/django_etebase/drf_msgpack/renderers.py +++ /dev/null @@ -1,15 +0,0 @@ -import msgpack - -from rest_framework.renderers import BaseRenderer - - -class MessagePackRenderer(BaseRenderer): - media_type = "application/msgpack" - format = "msgpack" - render_style = "binary" - charset = None - - def render(self, data, media_type=None, renderer_context=None): - if data is None: - return b"" - return msgpack.packb(data, use_bin_type=True) diff --git a/django_etebase/drf_msgpack/views.py b/django_etebase/drf_msgpack/views.py deleted file mode 100644 index 91ea44a..0000000 --- a/django_etebase/drf_msgpack/views.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.shortcuts import render - -# Create your views here. diff --git a/django_etebase/exceptions.py b/django_etebase/exceptions.py deleted file mode 100644 index 437a71c..0000000 --- a/django_etebase/exceptions.py +++ /dev/null @@ -1,12 +0,0 @@ -from rest_framework import serializers, status - - -class EtebaseValidationError(serializers.ValidationError): - def __init__(self, code, detail, status_code=status.HTTP_400_BAD_REQUEST): - super().__init__( - { - "code": code, - "detail": detail, - } - ) - self.status_code = status_code diff --git a/django_etebase/parsers.py b/django_etebase/parsers.py deleted file mode 100644 index ed1e713..0000000 --- a/django_etebase/parsers.py +++ /dev/null @@ -1,14 +0,0 @@ -from rest_framework.parsers import FileUploadParser - - -class ChunkUploadParser(FileUploadParser): - """ - Parser for chunk upload data. - """ - - def get_filename(self, stream, media_type, parser_context): - """ - Detects the uploaded file name. - """ - view = parser_context["view"] - return parser_context["kwargs"][view.lookup_field] diff --git a/django_etebase/permissions.py b/django_etebase/permissions.py deleted file mode 100644 index 3c77d06..0000000 --- a/django_etebase/permissions.py +++ /dev/null @@ -1,93 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -from rest_framework import permissions -from django_etebase.models import Collection, AccessLevels - - -def is_collection_admin(collection, user): - member = collection.members.filter(user=user).first() - return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) - - -class IsCollectionAdmin(permissions.BasePermission): - """ - Custom permission to only allow owners of a collection to view it - """ - - message = { - "detail": "Only collection admins can perform this operation.", - "code": "admin_access_required", - } - - def has_permission(self, request, view): - collection_uid = view.kwargs["collection_uid"] - try: - collection = view.get_collection_queryset().get(main_item__uid=collection_uid) - return is_collection_admin(collection, request.user) - except Collection.DoesNotExist: - # If the collection does not exist, we want to 404 later, not permission denied. - return True - - -class IsCollectionAdminOrReadOnly(permissions.BasePermission): - """ - Custom permission to only allow owners of a collection to edit it - """ - - message = { - "detail": "Only collection admins can edit collections.", - "code": "admin_access_required", - } - - def has_permission(self, request, view): - collection_uid = view.kwargs.get("collection_uid", None) - - # Allow creating new collections - if collection_uid is None: - return True - - try: - collection = view.get_collection_queryset().get(main_item__uid=collection_uid) - if request.method in permissions.SAFE_METHODS: - return True - - return is_collection_admin(collection, request.user) - except Collection.DoesNotExist: - # If the collection does not exist, we want to 404 later, not permission denied. - return True - - -class HasWriteAccessOrReadOnly(permissions.BasePermission): - """ - Custom permission to restrict write - """ - - message = { - "detail": "You need write access to write to this collection", - "code": "no_write_access", - } - - def has_permission(self, request, view): - collection_uid = view.kwargs["collection_uid"] - try: - collection = view.get_collection_queryset().get(main_item__uid=collection_uid) - if request.method in permissions.SAFE_METHODS: - return True - else: - member = collection.members.get(user=request.user) - return member.accessLevel != AccessLevels.READ_ONLY - except Collection.DoesNotExist: - # If the collection does not exist, we want to 404 later, not permission denied. - return True diff --git a/django_etebase/renderers.py b/django_etebase/renderers.py deleted file mode 100644 index 0d359d3..0000000 --- a/django_etebase/renderers.py +++ /dev/null @@ -1,19 +0,0 @@ -from rest_framework.utils.encoders import JSONEncoder as DRFJSONEncoder -from rest_framework.renderers import JSONRenderer as DRFJSONRenderer - -from .serializers import b64encode - - -class JSONEncoder(DRFJSONEncoder): - def default(self, obj): - if isinstance(obj, bytes) or isinstance(obj, memoryview): - return b64encode(obj) - return super().default(obj) - - -class JSONRenderer(DRFJSONRenderer): - """ - Renderer which serializes to JSON with support for our base64 - """ - - encoder_class = JSONEncoder diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py deleted file mode 100644 index 26ac5a7..0000000 --- a/django_etebase/serializers.py +++ /dev/null @@ -1,598 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -import base64 - -from django.core.files.base import ContentFile -from django.core import exceptions as django_exceptions -from django.contrib.auth import get_user_model -from django.db import IntegrityError, transaction -from rest_framework import serializers, status -from . import models -from .utils import get_user_queryset, create_user, CallbackContext - -from .exceptions import EtebaseValidationError - -User = get_user_model() - - -def process_revisions_for_item(item, revision_data): - chunks_objs = [] - chunks = revision_data.pop("chunks_relation") - - revision = models.CollectionItemRevision(**revision_data, item=item) - revision.validate_unique() # Verify there aren't any validation issues - - for chunk in chunks: - uid = chunk[0] - chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() - content = chunk[1] if len(chunk) > 1 else None - # If the chunk already exists we assume it's fine. Otherwise, we upload it. - if chunk_obj is None: - if content is not None: - chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) - chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) - chunk_obj.save() - else: - raise EtebaseValidationError("chunk_no_content", "Tried to create a new chunk without content") - - chunks_objs.append(chunk_obj) - - stoken = models.Stoken.objects.create() - revision.stoken = stoken - revision.save() - - for chunk in chunks_objs: - models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) - return revision - - -def b64encode(value): - return base64.urlsafe_b64encode(value).decode("ascii").strip("=") - - -def b64decode(data): - data += "=" * ((4 - len(data) % 4) % 4) - return base64.urlsafe_b64decode(data) - - -def b64decode_or_bytes(data): - if isinstance(data, bytes): - return data - else: - return b64decode(data) - - -class BinaryBase64Field(serializers.Field): - def to_representation(self, value): - return value - - def to_internal_value(self, data): - return b64decode_or_bytes(data) - - -class CollectionEncryptionKeyField(BinaryBase64Field): - def get_attribute(self, instance): - request = self.context.get("request", None) - if request is not None: - return instance.members.get(user=request.user).encryptionKey - return None - - -class CollectionTypeField(BinaryBase64Field): - def get_attribute(self, instance): - request = self.context.get("request", None) - if request is not None: - collection_type = instance.members.get(user=request.user).collectionType - return collection_type and collection_type.uid - return None - - -class UserSlugRelatedField(serializers.SlugRelatedField): - def get_queryset(self): - view = self.context.get("view", None) - return get_user_queryset(super().get_queryset(), context=CallbackContext(view.kwargs)) - - def __init__(self, **kwargs): - super().__init__(slug_field=User.USERNAME_FIELD, **kwargs) - - def to_internal_value(self, data): - return super().to_internal_value(data.lower()) - - -class ChunksField(serializers.RelatedField): - def to_representation(self, obj): - obj = obj.chunk - if self.context.get("prefetch") == "auto": - with open(obj.chunkFile.path, "rb") as f: - return (obj.uid, f.read()) - else: - return (obj.uid,) - - def to_internal_value(self, data): - content = data[1] if len(data) > 1 else None - if data[0] is None: - raise EtebaseValidationError("no_null", "null is not allowed") - return (data[0], b64decode_or_bytes(content) if content is not None else None) - - -class BetterErrorsMixin: - @property - def errors(self): - nice = [] - errors = super().errors - for error_type in errors: - if error_type == "non_field_errors": - nice.extend(self.flatten_errors(None, errors[error_type])) - else: - nice.extend(self.flatten_errors(error_type, errors[error_type])) - if nice: - return {"code": "field_errors", "detail": "Field validations failed.", "errors": nice} - return {} - - def flatten_errors(self, field_name, errors): - ret = [] - if isinstance(errors, dict): - for error_key in errors: - error = errors[error_key] - ret.extend(self.flatten_errors("{}.{}".format(field_name, error_key), error)) - else: - for error in errors: - if getattr(error, "messages", None): - message = error.messages[0] - else: - message = str(error) - ret.append( - { - "field": field_name, - "code": error.code, - "detail": message, - } - ) - return ret - - def transform_validation_error(self, prefix, err): - if hasattr(err, "error_dict"): - errors = self.flatten_errors(prefix, err.error_dict) - elif not hasattr(err, "message"): - errors = self.flatten_errors(prefix, err.error_list) - else: - raise EtebaseValidationError(err.code, err.message) - - raise serializers.ValidationError( - { - "code": "field_errors", - "detail": "Field validations failed.", - "errors": errors, - } - ) - - -class CollectionItemChunkSerializer(BetterErrorsMixin, serializers.ModelSerializer): - class Meta: - model = models.CollectionItemChunk - fields = ("uid", "chunkFile") - - -class CollectionItemRevisionSerializer(BetterErrorsMixin, serializers.ModelSerializer): - chunks = ChunksField( - source="chunks_relation", - queryset=models.RevisionChunkRelation.objects.all(), - style={"base_template": "input.html"}, - many=True, - ) - meta = BinaryBase64Field() - - class Meta: - model = models.CollectionItemRevision - fields = ("chunks", "meta", "uid", "deleted") - extra_kwargs = { - "uid": {"validators": []}, # We deal with it in the serializers - } - - -class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): - encryptionKey = BinaryBase64Field(required=False, default=None, allow_null=True) - etag = serializers.CharField(allow_null=True, write_only=True) - content = CollectionItemRevisionSerializer(many=False) - - class Meta: - model = models.CollectionItem - fields = ("uid", "version", "encryptionKey", "content", "etag") - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - validate_etag = self.context.get("validate_etag", False) - etag = validated_data.pop("etag") - revision_data = validated_data.pop("content") - uid = validated_data.pop("uid") - - Model = self.__class__.Meta.model - - with transaction.atomic(): - instance, created = Model.objects.get_or_create(uid=uid, defaults=validated_data) - cur_etag = instance.etag if not created else None - - # If we are trying to update an up to date item, abort early and consider it a success - if cur_etag == revision_data.get("uid"): - return instance - - if validate_etag and cur_etag != etag: - raise EtebaseValidationError( - "wrong_etag", - "Wrong etag. Expected {} got {}".format(cur_etag, etag), - status_code=status.HTTP_409_CONFLICT, - ) - - if not created: - # We don't have to use select_for_update here because the unique constraint on current guards against - # the race condition. But it's a good idea because it'll lock and wait rather than fail. - current_revision = instance.revisions.filter(current=True).select_for_update().first() - - # If we are just re-uploading the same revision, consider it a succes and return. - if current_revision.uid == revision_data.get("uid"): - return instance - - current_revision.current = None - current_revision.save() - - try: - process_revisions_for_item(instance, revision_data) - except django_exceptions.ValidationError as e: - self.transform_validation_error("content", e) - - return instance - - def update(self, instance, validated_data): - # We never update, we always update in the create method - raise NotImplementedError() - - -class CollectionItemDepSerializer(BetterErrorsMixin, serializers.ModelSerializer): - etag = serializers.CharField() - - class Meta: - model = models.CollectionItem - fields = ("uid", "etag") - - def validate(self, data): - item = self.__class__.Meta.model.objects.get(uid=data["uid"]) - etag = data["etag"] - if item.etag != etag: - raise EtebaseValidationError( - "wrong_etag", - "Wrong etag. Expected {} got {}".format(item.etag, etag), - status_code=status.HTTP_409_CONFLICT, - ) - - return data - - -class CollectionItemBulkGetSerializer(BetterErrorsMixin, serializers.ModelSerializer): - etag = serializers.CharField(required=False) - - class Meta: - model = models.CollectionItem - fields = ("uid", "etag") - - -class CollectionListMultiSerializer(BetterErrorsMixin, serializers.Serializer): - collectionTypes = serializers.ListField(child=BinaryBase64Field()) - - -class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): - collectionKey = CollectionEncryptionKeyField() - collectionType = CollectionTypeField() - accessLevel = serializers.SerializerMethodField("get_access_level_from_context") - stoken = serializers.CharField(read_only=True) - - item = CollectionItemSerializer(many=False, source="main_item") - - class Meta: - model = models.Collection - fields = ("item", "accessLevel", "collectionKey", "collectionType", "stoken") - - def get_access_level_from_context(self, obj): - request = self.context.get("request", None) - if request is not None: - return obj.members.get(user=request.user).accessLevel - return None - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - collection_key = validated_data.pop("collectionKey") - collection_type = validated_data.pop("collectionType") - - user = validated_data.get("owner") - main_item_data = validated_data.pop("main_item") - uid = main_item_data.get("uid") - etag = main_item_data.pop("etag") - revision_data = main_item_data.pop("content") - - instance = self.__class__.Meta.model(uid=uid, **validated_data) - - with transaction.atomic(): - if etag is not None: - raise EtebaseValidationError("bad_etag", "etag is not null") - - try: - instance.validate_unique() - except django_exceptions.ValidationError: - raise EtebaseValidationError( - "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT - ) - instance.save() - - main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) - - instance.main_item = main_item - instance.save() - - process_revisions_for_item(main_item, revision_data) - - collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) - - models.CollectionMember( - collection=instance, - stoken=models.Stoken.objects.create(), - user=user, - accessLevel=models.AccessLevels.ADMIN, - encryptionKey=collection_key, - collectionType=collection_type_obj, - ).save() - - return instance - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField( - source="user", - read_only=True, - style={"base_template": "input.html"}, - ) - - class Meta: - model = models.CollectionMember - fields = ("username", "accessLevel") - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - with transaction.atomic(): - # We only allow updating accessLevel - access_level = validated_data.pop("accessLevel") - if instance.accessLevel != access_level: - instance.stoken = models.Stoken.objects.create() - instance.accessLevel = access_level - instance.save() - - return instance - - -class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField( - source="user", - queryset=User.objects, - style={"base_template": "input.html"}, - ) - collection = serializers.CharField(source="collection.uid") - fromUsername = serializers.CharField(source="fromMember.user.username", read_only=True) - fromPubkey = BinaryBase64Field(source="fromMember.user.userinfo.pubkey", read_only=True) - signedEncryptionKey = BinaryBase64Field() - - class Meta: - model = models.CollectionInvitation - fields = ( - "username", - "uid", - "collection", - "signedEncryptionKey", - "accessLevel", - "fromUsername", - "fromPubkey", - "version", - ) - - def validate_user(self, value): - request = self.context["request"] - - if request.user.username == value.lower(): - raise EtebaseValidationError("no_self_invite", "Inviting yourself is not allowed") - return value - - def create(self, validated_data): - request = self.context["request"] - collection = validated_data.pop("collection") - - member = collection.members.get(user=request.user) - - with transaction.atomic(): - try: - return type(self).Meta.model.objects.create(**validated_data, fromMember=member) - except IntegrityError: - raise EtebaseValidationError("invitation_exists", "Invitation already exists") - - def update(self, instance, validated_data): - with transaction.atomic(): - instance.accessLevel = validated_data.pop("accessLevel") - instance.signedEncryptionKey = validated_data.pop("signedEncryptionKey") - instance.save() - - return instance - - -class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): - collectionType = BinaryBase64Field() - encryptionKey = BinaryBase64Field() - - def create(self, validated_data): - - with transaction.atomic(): - invitation = self.context["invitation"] - encryption_key = validated_data.get("encryptionKey") - collection_type = validated_data.pop("collectionType") - - user = invitation.user - collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) - - member = models.CollectionMember.objects.create( - collection=invitation.collection, - stoken=models.Stoken.objects.create(), - user=user, - accessLevel=invitation.accessLevel, - encryptionKey=encryption_key, - collectionType=collection_type_obj, - ) - - models.CollectionMemberRemoved.objects.filter( - user=invitation.user, collection=invitation.collection - ).delete() - - invitation.delete() - - return member - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class UserSerializer(BetterErrorsMixin, serializers.ModelSerializer): - pubkey = BinaryBase64Field(source="userinfo.pubkey") - encryptedContent = BinaryBase64Field(source="userinfo.encryptedContent") - - class Meta: - model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, "pubkey", "encryptedContent") - - -class UserInfoPubkeySerializer(BetterErrorsMixin, serializers.ModelSerializer): - pubkey = BinaryBase64Field() - - class Meta: - model = models.UserInfo - fields = ("pubkey",) - - -class UserSignupSerializer(BetterErrorsMixin, serializers.ModelSerializer): - class Meta: - model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) - extra_kwargs = { - "username": {"validators": []}, # We specifically validate in SignupSerializer - } - - -class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): - """Used both for creating new accounts and setting up existing ones for the first time. - When setting up existing ones the email is ignored." - """ - - user = UserSignupSerializer(many=False) - salt = BinaryBase64Field() - loginPubkey = BinaryBase64Field() - pubkey = BinaryBase64Field() - encryptedContent = BinaryBase64Field() - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - user_data = validated_data.pop("user") - - with transaction.atomic(): - view = self.context.get("view", None) - try: - user_queryset = get_user_queryset(User.objects.all(), context=CallbackContext(view.kwargs)) - instance = user_queryset.get(**{User.USERNAME_FIELD: user_data["username"].lower()}) - except User.DoesNotExist: - # Create the user and save the casing the user chose as the first name - try: - instance = create_user( - **user_data, - password=None, - first_name=user_data["username"], - context=CallbackContext(view.kwargs) - ) - instance.full_clean() - except EtebaseValidationError as e: - raise e - except django_exceptions.ValidationError as e: - self.transform_validation_error("user", e) - except Exception as e: - raise EtebaseValidationError("generic", str(e)) - - if hasattr(instance, "userinfo"): - raise EtebaseValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) - - models.UserInfo.objects.create(**validated_data, owner=instance) - - return instance - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationLoginChallengeSerializer(BetterErrorsMixin, serializers.Serializer): - username = serializers.CharField(required=True) - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationLoginSerializer(BetterErrorsMixin, serializers.Serializer): - response = BinaryBase64Field() - signature = BinaryBase64Field() - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer): - challenge = BinaryBase64Field() - host = serializers.CharField() - action = serializers.CharField() - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationChangePasswordInnerSerializer(AuthenticationLoginInnerSerializer): - loginPubkey = BinaryBase64Field() - encryptedContent = BinaryBase64Field() - - class Meta: - model = models.UserInfo - fields = ("loginPubkey", "encryptedContent") - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - with transaction.atomic(): - instance.loginPubkey = validated_data.pop("loginPubkey") - instance.encryptedContent = validated_data.pop("encryptedContent") - instance.save() - - return instance diff --git a/django_etebase/tests.py b/django_etebase/tests.py deleted file mode 100644 index 7ce503c..0000000 --- a/django_etebase/tests.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.test import TestCase - -# Create your tests here. diff --git a/django_etebase/token_auth/admin.py b/django_etebase/token_auth/admin.py deleted file mode 100644 index e69de29..0000000 diff --git a/django_etebase/token_auth/authentication.py b/django_etebase/token_auth/authentication.py deleted file mode 100644 index 7e84956..0000000 --- a/django_etebase/token_auth/authentication.py +++ /dev/null @@ -1,46 +0,0 @@ -from django.utils import timezone -from django.utils.translation import gettext_lazy as _ - -from rest_framework import exceptions -from rest_framework.authentication import TokenAuthentication as DRFTokenAuthentication - -from .models import AuthToken, get_default_expiry - - -AUTO_REFRESH = True -MIN_REFRESH_INTERVAL = 60 - - -class TokenAuthentication(DRFTokenAuthentication): - keyword = "Token" - model = AuthToken - - def authenticate_credentials(self, key): - msg = _("Invalid token.") - model = self.get_model() - try: - token = model.objects.select_related("user").get(key=key) - except model.DoesNotExist: - raise exceptions.AuthenticationFailed(msg) - - if not token.user.is_active: - raise exceptions.AuthenticationFailed(_("User inactive or deleted.")) - - if token.expiry is not None: - if token.expiry < timezone.now(): - token.delete() - raise exceptions.AuthenticationFailed(msg) - - if AUTO_REFRESH: - self.renew_token(token) - - return (token.user, token) - - def renew_token(self, auth_token): - current_expiry = auth_token.expiry - new_expiry = get_default_expiry() - # Throttle refreshing of token to avoid db writes - delta = (new_expiry - current_expiry).total_seconds() - if delta > MIN_REFRESH_INTERVAL: - auth_token.expiry = new_expiry - auth_token.save(update_fields=("expiry",)) diff --git a/django_etebase/urls.py b/django_etebase/urls.py deleted file mode 100644 index 01797c1..0000000 --- a/django_etebase/urls.py +++ /dev/null @@ -1,30 +0,0 @@ -from django.conf import settings -from django.conf.urls import include -from django.urls import path - -from rest_framework_nested import routers - -from django_etebase import views - -router = routers.DefaultRouter() -router.register(r"collection", views.CollectionViewSet) -router.register(r"authentication", views.AuthenticationViewSet, basename="authentication") -router.register(r"invitation/incoming", views.InvitationIncomingViewSet, basename="invitation_incoming") -router.register(r"invitation/outgoing", views.InvitationOutgoingViewSet, basename="invitation_outgoing") - -collections_router = routers.NestedSimpleRouter(router, r"collection", lookup="collection") -collections_router.register(r"item", views.CollectionItemViewSet, basename="collection_item") -collections_router.register(r"member", views.CollectionMemberViewSet, basename="collection_member") - -item_router = routers.NestedSimpleRouter(collections_router, r"item", lookup="collection_item") -item_router.register(r"chunk", views.CollectionItemChunkViewSet, basename="collection_items_chunk") - -if settings.DEBUG: - router.register(r"test/authentication", views.TestAuthenticationViewSet, basename="test_authentication") - -app_name = "django_etebase" -urlpatterns = [ - path("v1/", include(router.urls)), - path("v1/", include(collections_router.urls)), - path("v1/", include(item_router.urls)), -] diff --git a/django_etebase/views.py b/django_etebase/views.py deleted file mode 100644 index 5a03aa4..0000000 --- a/django_etebase/views.py +++ /dev/null @@ -1,861 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -import msgpack - -from django.conf import settings -from django.contrib.auth import get_user_model, user_logged_in, user_logged_out -from django.core.exceptions import PermissionDenied -from django.db import transaction, IntegrityError -from django.db.models import Q -from django.http import HttpResponseBadRequest, HttpResponse, Http404 -from django.shortcuts import get_object_or_404 - -from rest_framework import status -from rest_framework import viewsets -from rest_framework.decorators import action as action_decorator -from rest_framework.response import Response -from rest_framework.parsers import JSONParser, FormParser, MultiPartParser -from rest_framework.renderers import BrowsableAPIRenderer -from rest_framework.exceptions import AuthenticationFailed -from rest_framework.permissions import IsAuthenticated - -import nacl.encoding -import nacl.signing -import nacl.secret -import nacl.hash - -from .sendfile import sendfile -from .token_auth.models import AuthToken - -from .drf_msgpack.parsers import MessagePackParser -from .drf_msgpack.renderers import MessagePackRenderer - -from . import app_settings, permissions -from .renderers import JSONRenderer -from .models import ( - Collection, - CollectionItem, - CollectionItemRevision, - CollectionMember, - CollectionMemberRemoved, - CollectionInvitation, - Stoken, - UserInfo, -) -from .serializers import ( - AuthenticationChangePasswordInnerSerializer, - AuthenticationSignupSerializer, - AuthenticationLoginChallengeSerializer, - AuthenticationLoginSerializer, - AuthenticationLoginInnerSerializer, - CollectionSerializer, - CollectionItemSerializer, - CollectionItemBulkGetSerializer, - CollectionItemDepSerializer, - CollectionItemRevisionSerializer, - CollectionItemChunkSerializer, - CollectionListMultiSerializer, - CollectionMemberSerializer, - CollectionInvitationSerializer, - InvitationAcceptSerializer, - UserInfoPubkeySerializer, - UserSerializer, -) -from .utils import get_user_queryset, CallbackContext -from .exceptions import EtebaseValidationError -from .parsers import ChunkUploadParser -from .signals import user_signed_up - -User = get_user_model() - - -def msgpack_encode(content): - return msgpack.packb(content, use_bin_type=True) - - -def msgpack_decode(content): - return msgpack.unpackb(content, raw=False) - - -class BaseViewSet(viewsets.ModelViewSet): - authentication_classes = tuple(app_settings.API_AUTHENTICATORS) - permission_classes = tuple(app_settings.API_PERMISSIONS) - renderer_classes = [JSONRenderer, MessagePackRenderer] + ([BrowsableAPIRenderer] if settings.DEBUG else []) - parser_classes = [JSONParser, MessagePackParser, FormParser, MultiPartParser] - stoken_annotation = None - - def get_serializer_class(self): - serializer_class = self.serializer_class - - if self.request.method == "PUT": - serializer_class = getattr(self, "serializer_update_class", serializer_class) - - return serializer_class - - def get_collection_queryset(self, queryset=Collection.objects): - user = self.request.user - return queryset.filter(members__user=user) - - def get_stoken_obj_id(self, request): - return request.GET.get("stoken", None) - - def get_stoken_obj(self, request): - stoken = self.get_stoken_obj_id(request) - - if stoken is not None: - try: - return Stoken.objects.get(uid=stoken) - except Stoken.DoesNotExist: - raise EtebaseValidationError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) - - return None - - def filter_by_stoken(self, request, queryset): - stoken_rev = self.get_stoken_obj(request) - - queryset = queryset.annotate(max_stoken=self.stoken_annotation).order_by("max_stoken") - - if stoken_rev is not None: - queryset = queryset.filter(max_stoken__gt=stoken_rev.id) - - return queryset, stoken_rev - - def get_queryset_stoken(self, queryset): - maxid = -1 - for row in queryset: - rowmaxid = getattr(row, "max_stoken") or -1 - maxid = max(maxid, rowmaxid) - new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid) - - return new_stoken or None - - def filter_by_stoken_and_limit(self, request, queryset): - limit = int(request.GET.get("limit", 50)) - - queryset, stoken_rev = self.filter_by_stoken(request, queryset) - - result = list(queryset[: limit + 1]) - if len(result) < limit + 1: - done = True - else: - done = False - result = result[:-1] - - new_stoken_obj = self.get_queryset_stoken(result) or stoken_rev - - return result, new_stoken_obj, done - - # Change how our list works by default - def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset() - serializer = self.get_serializer(queryset, many=True) - - ret = { - "data": serializer.data, - "done": True, # we always return all the items, so it's always done - } - - return Response(ret) - - -class CollectionViewSet(BaseViewSet): - allowed_methods = ["GET", "POST"] - permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly,) - queryset = Collection.objects.all() - serializer_class = CollectionSerializer - lookup_field = "uid" - lookup_url_kwarg = "uid" - stoken_annotation = Collection.stoken_annotation - - def get_queryset(self, queryset=None): - if queryset is None: - queryset = type(self).queryset - return self.get_collection_queryset(queryset) - - def get_serializer_context(self): - context = super().get_serializer_context() - prefetch = self.request.query_params.get("prefetch", "auto") - context.update({"request": self.request, "prefetch": prefetch}) - return context - - def destroy(self, request, uid=None, *args, **kwargs): - # FIXME: implement - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def partial_update(self, request, uid=None, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def update(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def create(self, request, *args, **kwargs): - serializer = self.get_serializer(data=request.data) - serializer.is_valid(raise_exception=True) - serializer.save(owner=self.request.user) - - return Response({}, status=status.HTTP_201_CREATED) - - def list(self, request, *args, **kwargs): - queryset = self.get_queryset() - return self.list_common(request, queryset, *args, **kwargs) - - @action_decorator(detail=False, methods=["POST"]) - def list_multi(self, request, *args, **kwargs): - serializer = CollectionListMultiSerializer(data=request.data) - serializer.is_valid(raise_exception=True) - - collection_types = serializer.validated_data["collectionTypes"] - - queryset = self.get_queryset() - # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") - queryset = queryset.filter( - Q(members__collectionType__uid__in=collection_types) | Q(members__collectionType__isnull=True) - ) - - return self.list_common(request, queryset, *args, **kwargs) - - def list_common(self, request, queryset, *args, **kwargs): - result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - - serializer = self.get_serializer(result, many=True) - - ret = { - "data": serializer.data, - "stoken": new_stoken, - "done": done, - } - - stoken_obj = self.get_stoken_obj(request) - if stoken_obj is not None: - # FIXME: honour limit? (the limit should be combined for data and this because of stoken) - remed_qs = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) - if not ret["done"]: - # We only filter by the new_stoken if we are not done. This is because if we are done, the new stoken - # can point to the most recent collection change rather than most recent removed membership. - remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) - - remed = remed_qs.values_list("collection__uid", flat=True) - if len(remed) > 0: - ret["removedMemberships"] = [{"uid": x} for x in remed] - - return Response(ret) - - -class CollectionItemViewSet(BaseViewSet): - allowed_methods = ["GET", "POST", "PUT"] - permission_classes = BaseViewSet.permission_classes + (permissions.HasWriteAccessOrReadOnly,) - queryset = CollectionItem.objects.all() - serializer_class = CollectionItemSerializer - lookup_field = "uid" - stoken_annotation = CollectionItem.stoken_annotation - - def get_queryset(self): - collection_uid = self.kwargs["collection_uid"] - try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) - except Collection.DoesNotExist: - raise Http404("Collection does not exist") - # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') - queryset = type(self).queryset.filter(collection__pk=collection.pk, revisions__current=True) - - return queryset - - def get_serializer_context(self): - context = super().get_serializer_context() - prefetch = self.request.query_params.get("prefetch", "auto") - context.update({"request": self.request, "prefetch": prefetch}) - return context - - def create(self, request, collection_uid=None, *args, **kwargs): - # We create using batch and transaction - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def destroy(self, request, collection_uid=None, uid=None, *args, **kwargs): - # We can't have destroy because we need to get data from the user (in the body) such as hmac. - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def update(self, request, collection_uid=None, uid=None, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def partial_update(self, request, collection_uid=None, uid=None, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset() - - if not self.request.query_params.get("withCollection", False): - queryset = queryset.filter(parent__isnull=True) - - result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - - serializer = self.get_serializer(result, many=True) - - ret = { - "data": serializer.data, - "stoken": new_stoken, - "done": done, - } - return Response(ret) - - @action_decorator(detail=True, methods=["GET"]) - def revision(self, request, collection_uid=None, uid=None, *args, **kwargs): - col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - item = get_object_or_404(col.items, uid=uid) - - limit = int(request.GET.get("limit", 50)) - iterator = request.GET.get("iterator", None) - - queryset = item.revisions.order_by("-id") - - if iterator is not None: - iterator = get_object_or_404(queryset, uid=iterator) - queryset = queryset.filter(id__lt=iterator.id) - - result = list(queryset[: limit + 1]) - if len(result) < limit + 1: - done = True - else: - done = False - result = result[:-1] - - serializer = CollectionItemRevisionSerializer(result, context=self.get_serializer_context(), many=True) - - iterator = serializer.data[-1]["uid"] if len(result) > 0 else None - - ret = { - "data": serializer.data, - "iterator": iterator, - "done": done, - } - return Response(ret) - - # FIXME: rename to something consistent with what the clients have - maybe list_updates? - @action_decorator(detail=False, methods=["POST"]) - def fetch_updates(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset() - - serializer = CollectionItemBulkGetSerializer(data=request.data, many=True) - serializer.is_valid(raise_exception=True) - # FIXME: make configurable? - item_limit = 200 - - if len(serializer.validated_data) > item_limit: - content = {"code": "too_many_items", "detail": "Request has too many items. Limit: {}".format(item_limit)} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - queryset, stoken_rev = self.filter_by_stoken(request, queryset) - - uids, etags = zip(*[(item["uid"], item.get("etag")) for item in serializer.validated_data]) - revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) - queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) - - new_stoken_obj = self.get_queryset_stoken(queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - stoken = stoken_rev and getattr(stoken_rev, "uid", None) - new_stoken = new_stoken or stoken - - serializer = self.get_serializer(queryset, many=True) - - ret = { - "data": serializer.data, - "stoken": new_stoken, - "done": True, # we always return all the items, so it's always done - } - return Response(ret) - - @action_decorator(detail=False, methods=["POST"]) - def batch(self, request, collection_uid=None, *args, **kwargs): - return self.transaction(request, collection_uid, validate_etag=False) - - @action_decorator(detail=False, methods=["POST"]) - def transaction(self, request, collection_uid=None, validate_etag=True, *args, **kwargs): - stoken = request.GET.get("stoken", None) - with transaction.atomic(): # We need this for locking on the collection object - collection_object = get_object_or_404( - self.get_collection_queryset(Collection.objects).select_for_update(), # Lock writes on the collection - uid=collection_uid, - ) - - if stoken is not None and stoken != collection_object.stoken: - content = {"code": "stale_stoken", "detail": "Stoken is too old"} - return Response(content, status=status.HTTP_409_CONFLICT) - - items = request.data.get("items") - deps = request.data.get("deps", None) - # FIXME: It should just be one serializer - context = self.get_serializer_context() - context.update({"validate_etag": validate_etag}) - serializer = self.get_serializer_class()(data=items, context=context, many=True) - deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) - - ser_valid = serializer.is_valid() - deps_ser_valid = deps is None or deps_serializer.is_valid() - if ser_valid and deps_ser_valid: - items = serializer.save(collection=collection_object) - - ret = {} - return Response(ret, status=status.HTTP_200_OK) - - return Response( - { - "items": serializer.errors, - "deps": deps_serializer.errors if deps is not None else [], - }, - status=status.HTTP_409_CONFLICT, - ) - - -class CollectionItemChunkViewSet(viewsets.ViewSet): - allowed_methods = ["GET", "PUT"] - authentication_classes = BaseViewSet.authentication_classes - permission_classes = BaseViewSet.permission_classes - renderer_classes = BaseViewSet.renderer_classes - parser_classes = (ChunkUploadParser,) - serializer_class = CollectionItemChunkSerializer - lookup_field = "uid" - - def get_serializer_class(self): - return self.serializer_class - - def get_collection_queryset(self, queryset=Collection.objects): - user = self.request.user - return queryset.filter(members__user=user) - - def update(self, request, *args, collection_uid=None, collection_item_uid=None, uid=None, **kwargs): - col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) - # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) - - data = { - "uid": uid, - "chunkFile": request.data["file"], - } - - serializer = self.get_serializer_class()(data=data) - serializer.is_valid(raise_exception=True) - try: - serializer.save(collection=col) - except IntegrityError: - return Response( - {"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT - ) - - return Response({}, status=status.HTTP_201_CREATED) - - @action_decorator(detail=True, methods=["GET"]) - def download(self, request, collection_uid=None, collection_item_uid=None, uid=None, *args, **kwargs): - col = get_object_or_404(self.get_collection_queryset(), uid=collection_uid) - chunk = get_object_or_404(col.chunks, uid=uid) - - filename = chunk.chunkFile.path - return sendfile(request, filename) - - -class CollectionMemberViewSet(BaseViewSet): - allowed_methods = ["GET", "PUT", "DELETE"] - our_base_permission_classes = BaseViewSet.permission_classes - permission_classes = our_base_permission_classes + (permissions.IsCollectionAdmin,) - queryset = CollectionMember.objects.all() - serializer_class = CollectionMemberSerializer - lookup_field = f"user__{User.USERNAME_FIELD}__iexact" - lookup_url_kwarg = "username" - stoken_annotation = CollectionMember.stoken_annotation - - # FIXME: need to make sure that there's always an admin, and maybe also don't let an owner remove adm access - # (if we want to transfer, we need to do that specifically) - - def get_queryset(self, queryset=None): - collection_uid = self.kwargs["collection_uid"] - try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) - except Collection.DoesNotExist: - raise Http404("Collection does not exist") - - if queryset is None: - queryset = type(self).queryset - - return queryset.filter(collection=collection) - - # We override this method because we expect the stoken to be called iterator - def get_stoken_obj_id(self, request): - return request.GET.get("iterator", None) - - def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset().order_by("id") - result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - serializer = self.get_serializer(result, many=True) - - ret = { - "data": serializer.data, - "iterator": new_stoken, # Here we call it an iterator, it's only stoken for collection/items - "done": done, - } - - return Response(ret) - - def create(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - # FIXME: block leaving if we are the last admins - should be deleted / assigned in this case depending if there - # are other memebers. - def perform_destroy(self, instance): - instance.revoke() - - @action_decorator(detail=False, methods=["POST"], permission_classes=our_base_permission_classes) - def leave(self, request, collection_uid=None, *args, **kwargs): - collection_uid = self.kwargs["collection_uid"] - col = get_object_or_404(self.get_collection_queryset(Collection.objects), uid=collection_uid) - - member = col.members.get(user=request.user) - self.perform_destroy(member) - - return Response({}) - - -class InvitationBaseViewSet(BaseViewSet): - queryset = CollectionInvitation.objects.all() - serializer_class = CollectionInvitationSerializer - lookup_field = "uid" - lookup_url_kwarg = "invitation_uid" - - def list(self, request, collection_uid=None, *args, **kwargs): - limit = int(request.GET.get("limit", 50)) - iterator = request.GET.get("iterator", None) - - queryset = self.get_queryset().order_by("id") - - if iterator is not None: - iterator = get_object_or_404(queryset, uid=iterator) - queryset = queryset.filter(id__gt=iterator.id) - - result = list(queryset[: limit + 1]) - if len(result) < limit + 1: - done = True - else: - done = False - result = result[:-1] - - serializer = self.get_serializer(result, many=True) - - iterator = serializer.data[-1]["uid"] if len(result) > 0 else None - - ret = { - "data": serializer.data, - "iterator": iterator, - "done": done, - } - - return Response(ret) - - -class InvitationOutgoingViewSet(InvitationBaseViewSet): - allowed_methods = ["GET", "POST", "PUT", "DELETE"] - - def get_queryset(self, queryset=None): - if queryset is None: - queryset = type(self).queryset - - return queryset.filter(fromMember__user=self.request.user) - - def create(self, request, *args, **kwargs): - serializer = self.get_serializer(data=request.data) - serializer.is_valid(raise_exception=True) - collection_uid = serializer.validated_data.get("collection", {}).get("uid") - - try: - collection = self.get_collection_queryset(Collection.objects).get(uid=collection_uid) - except Collection.DoesNotExist: - raise Http404("Collection does not exist") - - if request.user == serializer.validated_data.get("user"): - content = {"code": "self_invite", "detail": "Inviting yourself is invalid"} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - if not permissions.is_collection_admin(collection, request.user): - raise PermissionDenied( - {"code": "admin_access_required", "detail": "User is not an admin of this collection"} - ) - - serializer.save(collection=collection) - - return Response({}, status=status.HTTP_201_CREATED) - - @action_decorator(detail=False, allowed_methods=["GET"], methods=["GET"]) - def fetch_user_profile(self, request, *args, **kwargs): - username = request.GET.get("username") - kwargs = {User.USERNAME_FIELD: username.lower()} - user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(self.kwargs)), **kwargs) - user_info = get_object_or_404(UserInfo.objects.all(), owner=user) - serializer = UserInfoPubkeySerializer(user_info) - return Response(serializer.data) - - -class InvitationIncomingViewSet(InvitationBaseViewSet): - allowed_methods = ["GET", "DELETE"] - - def get_queryset(self, queryset=None): - if queryset is None: - queryset = type(self).queryset - - return queryset.filter(user=self.request.user) - - @action_decorator(detail=True, allowed_methods=["POST"], methods=["POST"]) - def accept(self, request, invitation_uid=None, *args, **kwargs): - invitation = get_object_or_404(self.get_queryset(), uid=invitation_uid) - context = self.get_serializer_context() - context.update({"invitation": invitation}) - - serializer = InvitationAcceptSerializer(data=request.data, context=context) - serializer.is_valid(raise_exception=True) - serializer.save() - return Response(status=status.HTTP_201_CREATED) - - -class AuthenticationViewSet(viewsets.ViewSet): - allowed_methods = ["POST"] - authentication_classes = BaseViewSet.authentication_classes - renderer_classes = BaseViewSet.renderer_classes - parser_classes = BaseViewSet.parser_classes - - def get_encryption_key(self, salt): - key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) - return nacl.hash.blake2b( - b"", - key=key, - salt=salt[: nacl.hash.BLAKE2B_SALTBYTES], - person=b"etebase-auth", - encoder=nacl.encoding.RawEncoder, - ) - - def get_queryset(self): - return get_user_queryset(User.objects.all(), CallbackContext(self.kwargs)) - - def get_serializer_context(self): - return {"request": self.request, "format": self.format_kwarg, "view": self} - - def login_response_data(self, user): - return { - "token": AuthToken.objects.create(user=user).key, - "user": UserSerializer(user).data, - } - - def list(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - @action_decorator(detail=False, methods=["POST"]) - def signup(self, request, *args, **kwargs): - serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) - serializer.is_valid(raise_exception=True) - user = serializer.save() - - user_signed_up.send(sender=user.__class__, request=request, user=user) - - data = self.login_response_data(user) - return Response(data, status=status.HTTP_201_CREATED) - - def get_login_user(self, username): - kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} - try: - user = self.get_queryset().get(**kwargs) - if not hasattr(user, "userinfo"): - raise AuthenticationFailed({"code": "user_not_init", "detail": "User not properly init"}) - return user - except User.DoesNotExist: - raise AuthenticationFailed({"code": "user_not_found", "detail": "User not found"}) - - def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): - from datetime import datetime - - username = validated_data.get("username") - user = self.get_login_user(username) - host = validated_data["host"] - challenge = validated_data["challenge"] - action = validated_data["action"] - - salt = bytes(user.userinfo.salt) - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = msgpack_decode(box.decrypt(challenge)) - now = int(datetime.now().timestamp()) - if action != expected_action: - content = {"code": "wrong_action", "detail": 'Expected "{}" but got something else'.format(expected_action)} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: - content = {"code": "challenge_expired", "detail": "Login challange has expired"} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif challenge_data["userId"] != user.id: - content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host.split(":", 1)[0] != request.get_host().split(":", 1)[0]: - detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) - content = {"code": "wrong_host", "detail": detail} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) - - try: - verify_key.verify(response_raw, signature) - except nacl.exceptions.BadSignatureError: - return Response( - {"code": "login_bad_signature", "detail": "Wrong password for user."}, - status=status.HTTP_401_UNAUTHORIZED, - ) - - return None - - @action_decorator(detail=False, methods=["GET"]) - def is_etebase(self, request, *args, **kwargs): - return Response({}, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"]) - def login_challenge(self, request, *args, **kwargs): - from datetime import datetime - - serializer = AuthenticationLoginChallengeSerializer(data=request.data) - serializer.is_valid(raise_exception=True) - username = serializer.validated_data.get("username") - user = self.get_login_user(username) - - salt = bytes(user.userinfo.salt) - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = { - "timestamp": int(datetime.now().timestamp()), - "userId": user.id, - } - challenge = box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder) - - ret = { - "salt": salt, - "challenge": challenge, - "version": user.userinfo.version, - } - return Response(ret, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"]) - def login(self, request, *args, **kwargs): - outer_serializer = AuthenticationLoginSerializer(data=request.data) - outer_serializer.is_valid(raise_exception=True) - - response_raw = outer_serializer.validated_data["response"] - response = msgpack_decode(response_raw) - signature = outer_serializer.validated_data["signature"] - - context = {"host": request.get_host()} - serializer = AuthenticationLoginInnerSerializer(data=response, context=context) - serializer.is_valid(raise_exception=True) - - bad_login_response = self.validate_login_request( - request, serializer.validated_data, response_raw, signature, "login" - ) - if bad_login_response is not None: - return bad_login_response - - username = serializer.validated_data.get("username") - user = self.get_login_user(username) - - data = self.login_response_data(user) - - user_logged_in.send(sender=user.__class__, request=request, user=user) - - return Response(data, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"], permission_classes=[IsAuthenticated]) - def logout(self, request, *args, **kwargs): - request.auth.delete() - user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) - return Response(status=status.HTTP_204_NO_CONTENT) - - @action_decorator(detail=False, methods=["POST"], permission_classes=BaseViewSet.permission_classes) - def change_password(self, request, *args, **kwargs): - outer_serializer = AuthenticationLoginSerializer(data=request.data) - outer_serializer.is_valid(raise_exception=True) - - response_raw = outer_serializer.validated_data["response"] - response = msgpack_decode(response_raw) - signature = outer_serializer.validated_data["signature"] - - context = {"host": request.get_host()} - serializer = AuthenticationChangePasswordInnerSerializer(request.user.userinfo, data=response, context=context) - serializer.is_valid(raise_exception=True) - - bad_login_response = self.validate_login_request( - request, serializer.validated_data, response_raw, signature, "changePassword" - ) - if bad_login_response is not None: - return bad_login_response - - serializer.save() - - return Response({}, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"], permission_classes=[IsAuthenticated]) - def dashboard_url(self, request, *args, **kwargs): - get_dashboard_url = app_settings.DASHBOARD_URL_FUNC - if get_dashboard_url is None: - raise EtebaseValidationError( - "not_supported", "This server doesn't have a user dashboard.", status_code=status.HTTP_400_BAD_REQUEST - ) - - ret = { - "url": get_dashboard_url(request, *args, **kwargs), - } - return Response(ret) - - -class TestAuthenticationViewSet(viewsets.ViewSet): - allowed_methods = ["POST"] - renderer_classes = BaseViewSet.renderer_classes - parser_classes = BaseViewSet.parser_classes - - def get_serializer_context(self): - return {"request": self.request, "format": self.format_kwarg, "view": self} - - def list(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - @action_decorator(detail=False, methods=["POST"]) - def reset(self, request, *args, **kwargs): - # Only run when in DEBUG mode! It's only used for tests - if not settings.DEBUG: - return HttpResponseBadRequest("Only allowed in debug mode.") - - with transaction.atomic(): - user_queryset = get_user_queryset(User.objects.all(), CallbackContext(self.kwargs)) - user = get_object_or_404(user_queryset, username=request.data.get("user").get("username")) - - # Only allow test users for extra safety - if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): - return HttpResponseBadRequest("Endpoint not allowed for user.") - - if hasattr(user, "userinfo"): - user.userinfo.delete() - - serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) - serializer.is_valid(raise_exception=True) - serializer.save() - - # Delete all of the journal data for this user for a clear test env - user.collection_set.all().delete() - user.collectionmember_set.all().delete() - user.incoming_invitations.all().delete() - - # FIXME: also delete chunk files!!! - - return HttpResponse() diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index a13cc51..1c262c8 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -18,16 +18,14 @@ from fastapi import APIRouter, Depends, status, Request from fastapi.security import APIKeyHeader from django_etebase import app_settings, models -from django_etebase.exceptions import EtebaseValidationError from django_etebase.models import UserInfo from django_etebase.signals import user_signed_up from django_etebase.token_auth.models import AuthToken from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user, get_user_queryset, CallbackContext -from django_etebase.views import msgpack_encode, msgpack_decode from .exceptions import AuthenticationFailed, transform_validation_error, HttpError from .msgpack import MsgpackRoute -from .utils import BaseModel, permission_responses +from .utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode User = get_user_model() token_scheme = APIKeyHeader(name="Authorization") @@ -293,7 +291,7 @@ def signup_save(data: SignupIn, request: Request) -> User: context=CallbackContext(request.path_params), ) instance.full_clean() - except EtebaseValidationError as e: + except HttpError as e: raise e except django_exceptions.ValidationError as e: transform_validation_error("user", e) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 165163a..3473fa0 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -1,5 +1,6 @@ import dataclasses import typing as t +import msgpack from fastapi import status, Query, Depends from pydantic import BaseModel as PyBaseModel @@ -44,6 +45,14 @@ def is_collection_admin(collection, user): return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) +def msgpack_encode(content): + return msgpack.packb(content, use_bin_type=True) + + +def msgpack_decode(content): + return msgpack.unpackb(content, raw=False) + + PERMISSIONS_READ = [Depends(x) for x in app_settings.API_PERMISSIONS_READ] PERMISSIONS_READWRITE = PERMISSIONS_READ + [Depends(x) for x in app_settings.API_PERMISSIONS_WRITE] diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 46ad3c9..5d57ec0 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -53,8 +53,6 @@ INSTALLED_APPS = [ "django.contrib.sessions", "django.contrib.messages", "django.contrib.staticfiles", - "corsheaders", - "rest_framework", "myauth.apps.MyauthConfig", "django_etebase.apps.DjangoEtebaseConfig", "django_etebase.token_auth.apps.TokenAuthConfig", @@ -63,7 +61,6 @@ INSTALLED_APPS = [ MIDDLEWARE = [ "django.middleware.security.SecurityMiddleware", "django.contrib.sessions.middleware.SessionMiddleware", - "corsheaders.middleware.CorsMiddleware", "django.middleware.common.CommonMiddleware", "django.middleware.csrf.CsrfViewMiddleware", "django.contrib.auth.middleware.AuthenticationMiddleware", @@ -124,9 +121,6 @@ USE_L10N = True USE_TZ = True -# Cors -CORS_ORIGIN_ALLOW_ALL = True - # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/3.0/howto/static-files/ diff --git a/etebase_server/urls.py b/etebase_server/urls.py index 443763d..7cf5a60 100644 --- a/etebase_server/urls.py +++ b/etebase_server/urls.py @@ -1,7 +1,7 @@ import os from django.conf import settings -from django.conf.urls import include, url +from django.conf.urls import url from django.contrib import admin from django.urls import path, re_path from django.views.generic import TemplateView @@ -9,15 +9,11 @@ from django.views.static import serve from django.contrib.staticfiles import finders urlpatterns = [ - url(r"^api/", include("django_etebase.urls")), url(r"^admin/", admin.site.urls), path("", TemplateView.as_view(template_name="success.html")), ] if settings.DEBUG: - urlpatterns += [ - url(r"^api-auth/", include("rest_framework.urls", namespace="rest_framework")), - ] def serve_static(request, path): filename = finders.find(path) diff --git a/requirements.in/base.txt b/requirements.in/base.txt index ca8dd94..fee4a56 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -1,9 +1,5 @@ django -django-cors-headers -djangorestframework -drf-nested-routers msgpack -psycopg2-binary pynacl fastapi -uvicorn \ No newline at end of file +uvicorn diff --git a/requirements.txt b/requirements.txt index 3d19eaf..cfce456 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,14 +7,10 @@ asgiref==3.3.1 # via django cffi==1.14.4 # via pynacl click==7.1.2 # via uvicorn -django-cors-headers==3.6.0 # via -r requirements.in/base.txt -django==3.1.4 # via -r requirements.in/base.txt, django-cors-headers, djangorestframework, drf-nested-routers -djangorestframework==3.12.2 # via -r requirements.in/base.txt, drf-nested-routers -drf-nested-routers==0.92.5 # via -r requirements.in/base.txt +django==3.1.4 # via -r requirements.in/base.txt fastapi==0.63.0 # via -r requirements.in/base.txt h11==0.11.0 # via uvicorn msgpack==1.0.2 # via -r requirements.in/base.txt -psycopg2-binary==2.8.6 # via -r requirements.in/base.txt pycparser==2.20 # via cffi pydantic==1.7.3 # via fastapi pynacl==1.4.0 # via -r requirements.in/base.txt From 2e9caf66f960db6d1177183643e6a4e88e3eba1c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 14:04:07 +0200 Subject: [PATCH 428/511] Remove deprecated settings. --- django_etebase/app_settings.py | 28 ---------------------------- 1 file changed, 28 deletions(-) diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings.py index c1e8dc9..90225a6 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings.py @@ -31,17 +31,6 @@ class AppSettings: return getattr(settings, self.prefix + name, dflt) - @cached_property - def API_PERMISSIONS(self): # pylint: disable=invalid-name - """ - Deprecated. Do not use. - """ - perms = self._setting("API_PERMISSIONS", ("rest_framework.permissions.IsAuthenticated",)) - ret = [] - for perm in perms: - ret.append(self.import_from_str(perm)) - return ret - @cached_property def API_PERMISSIONS_READ(self): # pylint: disable=invalid-name perms = self._setting("API_PERMISSIONS_READ", tuple()) @@ -58,23 +47,6 @@ class AppSettings: ret.append(self.import_from_str(perm)) return ret - @cached_property - def API_AUTHENTICATORS(self): # pylint: disable=invalid-name - """ - Deprecated. Do not use. - """ - perms = self._setting( - "API_AUTHENTICATORS", - ( - "rest_framework.authentication.TokenAuthentication", - "rest_framework.authentication.SessionAuthentication", - ), - ) - ret = [] - for perm in perms: - ret.append(self.import_from_str(perm)) - return ret - @cached_property def GET_USER_QUERYSET_FUNC(self): # pylint: disable=invalid-name get_user_queryset = self._setting("GET_USER_QUERYSET_FUNC", None) From c918d3ed076a799e8e5bced6f48dce8f1de5d5e2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 14:26:44 +0200 Subject: [PATCH 429/511] Add base64 utils. --- etebase_fastapi/utils.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 3473fa0..5e45db7 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -1,6 +1,7 @@ import dataclasses import typing as t import msgpack +import base64 from fastapi import status, Query, Depends from pydantic import BaseModel as PyBaseModel @@ -53,6 +54,15 @@ def msgpack_decode(content): return msgpack.unpackb(content, raw=False) +def b64encode(value): + return base64.urlsafe_b64encode(value).decode("ascii").strip("=") + + +def b64decode(data): + data += "=" * ((4 - len(data) % 4) % 4) + return base64.urlsafe_b64decode(data) + + PERMISSIONS_READ = [Depends(x) for x in app_settings.API_PERMISSIONS_READ] PERMISSIONS_READWRITE = PERMISSIONS_READ + [Depends(x) for x in app_settings.API_PERMISSIONS_WRITE] From 313dcf072119a52edd01bc514222185445a7edd7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 14:27:23 +0200 Subject: [PATCH 430/511] django_etebase utils: add optionl user to context. --- django_etebase/utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/django_etebase/utils.py b/django_etebase/utils.py index 1c8654b..09028c4 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -15,6 +15,7 @@ class CallbackContext: """Class for passing extra context to callbacks""" url_kwargs: t.Dict[str, t.Any] + user: t.Optional[User] def get_user_queryset(queryset, context: CallbackContext): From b3c170e10d6581af2c64f9451ca7ff35e5409cd0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 14:28:42 +0200 Subject: [PATCH 431/511] fix getting dashboard URL. --- etebase_fastapi/authentication.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 1c262c8..2f8395e 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -264,13 +264,12 @@ async def change_password(data: ChangePassword, request: Request, user: User = D @authentication_router.post("/dashboard_url/", responses=permission_responses) def dashboard_url(user: User = Depends(get_authenticated_user)): - # XXX-TOM get_dashboard_url = app_settings.DASHBOARD_URL_FUNC if get_dashboard_url is None: raise HttpError("not_supported", "This server doesn't have a user dashboard.") ret = { - "url": get_dashboard_url(request, *args, **kwargs), + "url": get_dashboard_url(CallbackContext(request.path_params, user=user)), } return ret From 65cd722616a5b7898ef559387463cdadce656482 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 14:27:23 +0200 Subject: [PATCH 432/511] django_etebase utils: add optionl user to context. --- django_etebase/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_etebase/utils.py b/django_etebase/utils.py index 09028c4..e46cbd0 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -15,7 +15,7 @@ class CallbackContext: """Class for passing extra context to callbacks""" url_kwargs: t.Dict[str, t.Any] - user: t.Optional[User] + user: t.Optional[User] = None def get_user_queryset(queryset, context: CallbackContext): From c1f171bde0ca3b1a908ae30f2fd24efd83914f54 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 14:47:41 +0200 Subject: [PATCH 433/511] Change how we create applications. --- etebase_fastapi/main.py | 52 +++++++++++++++++++++-------------------- etebase_server/asgi.py | 4 +++- 2 files changed, 30 insertions(+), 26 deletions(-) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index 2c10854..706081a 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -12,28 +12,30 @@ from .member import member_router from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse -app = FastAPI() -VERSION = "v1" -BASE_PATH = f"/api/{VERSION}" -COLLECTION_UID_MARKER = "{collection_uid}" -app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication", tags=["authentication"]) -app.include_router(collection_router, prefix=f"{BASE_PATH}/collection", tags=["collection"]) -app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["item"]) -app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["member"]) -app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"]) -app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"]) - -if settings.DEBUG: - from etebase_fastapi.test_reset_view import test_reset_view_router - - app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") - -app.add_middleware( - CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] -) -app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) - - -@app.exception_handler(CustomHttpException) -async def custom_exception_handler(request: Request, exc: CustomHttpException): - return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) +def create_application(prefix=""): + app = FastAPI() + VERSION = "v1" + BASE_PATH = f"{prefix}/api/{VERSION}" + COLLECTION_UID_MARKER = "{collection_uid}" + app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication", tags=["authentication"]) + app.include_router(collection_router, prefix=f"{BASE_PATH}/collection", tags=["collection"]) + app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["item"]) + app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["member"]) + app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"]) + app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"]) + + if settings.DEBUG: + from etebase_fastapi.test_reset_view import test_reset_view_router + + app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") + + app.add_middleware( + CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] + ) + app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) + + + @app.exception_handler(CustomHttpException) + async def custom_exception_handler(request: Request, exc: CustomHttpException): + return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) + return app diff --git a/etebase_server/asgi.py b/etebase_server/asgi.py index 92fad1c..25dbf77 100644 --- a/etebase_server/asgi.py +++ b/etebase_server/asgi.py @@ -7,7 +7,9 @@ django_application = get_asgi_application() def create_application(): - from etebase_fastapi.main import app + from etebase_fastapi.main import create_application + + app = create_application() app.mount("/", django_application) From 50f89c48e27d0d6d5aecac22d491e53e130a0bba Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 15:07:18 +0200 Subject: [PATCH 434/511] Dashboard url: fix getting dashboard url. --- etebase_fastapi/authentication.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 2f8395e..04aec31 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -263,7 +263,7 @@ async def change_password(data: ChangePassword, request: Request, user: User = D @authentication_router.post("/dashboard_url/", responses=permission_responses) -def dashboard_url(user: User = Depends(get_authenticated_user)): +def dashboard_url(request: Request, user: User = Depends(get_authenticated_user)): get_dashboard_url = app_settings.DASHBOARD_URL_FUNC if get_dashboard_url is None: raise HttpError("not_supported", "This server doesn't have a user dashboard.") From ca7f2ec73cb559f0875e580b81a1e0f18b06c21a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 15:08:03 +0200 Subject: [PATCH 435/511] When converting from ORM convert binary fields to bytes. The problem is that some ORMs return memoryview which are more efficient but are not supported by pydantic at the moment. --- etebase_fastapi/authentication.py | 9 +++++---- etebase_fastapi/collection.py | 6 +++--- etebase_fastapi/invitation.py | 8 ++++++-- 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 04aec31..2f5a2f1 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -25,7 +25,7 @@ from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user, get_user_queryset, CallbackContext from .exceptions import AuthenticationFailed, transform_validation_error, HttpError from .msgpack import MsgpackRoute -from .utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode +from .utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode User = get_user_model() token_scheme = APIKeyHeader(name="Authorization") @@ -63,7 +63,7 @@ class UserOut(BaseModel): @classmethod def from_orm(cls: t.Type["UserOut"], obj: User) -> "UserOut": - return cls(pubkey=obj.userinfo.pubkey, encryptedContent=obj.userinfo.encryptedContent) + return cls(pubkey=bytes(obj.userinfo.pubkey), encryptedContent=bytes(obj.userinfo.encryptedContent)) class LoginOut(BaseModel): @@ -228,14 +228,15 @@ async def is_etebase(): @authentication_router.post("/login_challenge/", response_model=LoginChallengeOut) async def login_challenge(user: User = Depends(get_login_user)): - enc_key = get_encryption_key(user.userinfo.salt) + salt = bytes(user.userinfo.salt) + enc_key = get_encryption_key(salt) box = nacl.secret.SecretBox(enc_key) challenge_data = { "timestamp": int(datetime.now().timestamp()), "userId": user.id, } challenge = bytes(box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder)) - return LoginChallengeOut(salt=user.userinfo.salt, challenge=challenge, version=user.userinfo.version) + return LoginChallengeOut(salt=salt, challenge=challenge, version=user.userinfo.version) @authentication_router.post("/login/", response_model=LoginOut) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index e6c10c3..74730ff 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -58,7 +58,7 @@ class CollectionItemRevisionInOut(BaseModel): chunks.append((chunk_obj.uid, f.read())) else: chunks.append((chunk_obj.uid,)) - return cls(uid=obj.uid, meta=obj.meta, deleted=obj.deleted, chunks=chunks) + return cls(uid=obj.uid, meta=bytes(obj.meta), deleted=obj.deleted, chunks=chunks) class CollectionItemCommon(BaseModel): @@ -103,8 +103,8 @@ class CollectionOut(CollectionCommon): member: models.CollectionMember = obj.members.get(user=context.user) collection_type = member.collectionType ret = cls( - collectionType=collection_type and collection_type.uid, - collectionKey=member.encryptionKey, + collectionType=collection_type and bytes(collection_type.uid), + collectionKey=bytes(member.encryptionKey), accessLevel=member.accessLevel, stoken=obj.stoken, item=CollectionItemOut.from_orm_context(obj.main_item, context), diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 39460a9..9e731bc 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -32,6 +32,10 @@ class UserInfoOut(BaseModel): class Config: orm_mode = True + @classmethod + def from_orm(cls: t.Type["UserInfoOut"], obj: models.UserInfo) -> "UserInfoOut": + return cls(pubkey=bytes(obj.pubkey)) + class CollectionInvitationAcceptIn(BaseModel): collectionType: bytes @@ -69,8 +73,8 @@ class CollectionInvitationOut(CollectionInvitationCommon): username=obj.user.username, collection=obj.collection.uid, fromUsername=obj.fromMember.user.username, - fromPubkey=obj.fromMember.user.userinfo.pubkey, - signedEncryptionKey=obj.signedEncryptionKey, + fromPubkey=bytes(obj.fromMember.user.userinfo.pubkey), + signedEncryptionKey=bytes(obj.signedEncryptionKey), ) From 59e30ed9884990f33408f85cd1c18666e61d5507 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 15:17:13 +0200 Subject: [PATCH 436/511] Signup and logout: make sync. --- etebase_fastapi/authentication.py | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 2f5a2f1..df5dc62 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -250,10 +250,9 @@ async def login(data: Login, request: Request): @authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) -async def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): - await sync_to_async(auth_data.token.delete)() - # XXX-TOM - await sync_to_async(user_logged_out.send)(sender=auth_data.user.__class__, request=None, user=auth_data.user) +def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): + auth_data.token.delete() + user_logged_out.send(sender=auth_data.user.__class__, request=None, user=auth_data.user) @authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) @@ -306,9 +305,8 @@ def signup_save(data: SignupIn, request: Request) -> User: @authentication_router.post("/signup/", response_model=LoginOut, status_code=status.HTTP_201_CREATED) -async def signup(data: SignupIn, request: Request): - user = await sync_to_async(signup_save)(data, request) - # XXX-TOM - data = await sync_to_async(LoginOut.from_orm)(user) - await sync_to_async(user_signed_up.send)(sender=user.__class__, request=None, user=user) +def signup(data: SignupIn, request: Request): + user = signup_save(data, request) + data = LoginOut.from_orm(user) + user_signed_up.send(sender=user.__class__, request=None, user=user) return data From 1bca435d740f1c279efa2113e1ec9d59edc995b8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 15:26:34 +0200 Subject: [PATCH 437/511] Workaround typing issue. --- etebase_fastapi/utils.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 5e45db7..7280018 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -68,4 +68,7 @@ PERMISSIONS_READWRITE = PERMISSIONS_READ + [Depends(x) for x in app_settings.API response_model_dict = {"model": HttpErrorOut} -permission_responses = {401: response_model_dict, 403: response_model_dict} +permission_responses: t.Dict[t.Union[int, str], t.Dict[str, t.Any]] = { + 401: response_model_dict, + 403: response_model_dict, +} From 3e39aa88a15579090286cc8a859c6d0256652caa Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 15:27:29 +0200 Subject: [PATCH 438/511] Remove unused var. --- etebase_fastapi/authentication.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index df5dc62..9c66a7a 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -250,7 +250,7 @@ async def login(data: Login, request: Request): @authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) -def logout(request: Request, auth_data: AuthData = Depends(get_auth_data)): +def logout(auth_data: AuthData = Depends(get_auth_data)): auth_data.token.delete() user_logged_out.send(sender=auth_data.user.__class__, request=None, user=auth_data.user) From c2a2e710c9c9cca7b1bcc9b4523fbe095904c26b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 15:38:00 +0200 Subject: [PATCH 439/511] Move common dependencies to their own file. --- etebase_fastapi/authentication.py | 57 +-------------------- etebase_fastapi/collection.py | 21 +------- etebase_fastapi/dependencies.py | 82 +++++++++++++++++++++++++++++++ 3 files changed, 86 insertions(+), 74 deletions(-) create mode 100644 etebase_fastapi/dependencies.py diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 9c66a7a..eb54f68 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -1,4 +1,3 @@ -import dataclasses import typing as t from datetime import datetime from functools import cached_property @@ -13,33 +12,22 @@ from django.conf import settings from django.contrib.auth import get_user_model, user_logged_out, user_logged_in from django.core import exceptions as django_exceptions from django.db import transaction -from django.utils import timezone from fastapi import APIRouter, Depends, status, Request -from fastapi.security import APIKeyHeader from django_etebase import app_settings, models +from django_etebase.token_auth.models import AuthToken from django_etebase.models import UserInfo from django_etebase.signals import user_signed_up -from django_etebase.token_auth.models import AuthToken -from django_etebase.token_auth.models import get_default_expiry from django_etebase.utils import create_user, get_user_queryset, CallbackContext from .exceptions import AuthenticationFailed, transform_validation_error, HttpError from .msgpack import MsgpackRoute from .utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode +from .dependencies import AuthData, get_auth_data, get_authenticated_user User = get_user_model() -token_scheme = APIKeyHeader(name="Authorization") -AUTO_REFRESH = True -MIN_REFRESH_INTERVAL = 60 authentication_router = APIRouter(route_class=MsgpackRoute) -@dataclasses.dataclass(frozen=True) -class AuthData: - user: User - token: AuthToken - - class LoginChallengeIn(BaseModel): username: str @@ -115,47 +103,6 @@ class SignupIn(BaseModel): encryptedContent: bytes -def __renew_token(auth_token: AuthToken): - current_expiry = auth_token.expiry - new_expiry = get_default_expiry() - # Throttle refreshing of token to avoid db writes - delta = (new_expiry - current_expiry).total_seconds() - if delta > MIN_REFRESH_INTERVAL: - auth_token.expiry = new_expiry - auth_token.save(update_fields=("expiry",)) - - -@sync_to_async -def __get_authenticated_user(api_token: str): - api_token = api_token.split()[1] - try: - token: AuthToken = AuthToken.objects.select_related("user").get(key=api_token) - except AuthToken.DoesNotExist: - raise AuthenticationFailed(detail="Invalid token.") - if not token.user.is_active: - raise AuthenticationFailed(detail="User inactive or deleted.") - - if token.expiry is not None: - if token.expiry < timezone.now(): - token.delete() - raise AuthenticationFailed(detail="Invalid token.") - - if AUTO_REFRESH: - __renew_token(token) - - return token.user, token - - -async def get_auth_data(api_token: str = Depends(token_scheme)) -> AuthData: - user, token = await __get_authenticated_user(api_token) - return AuthData(user, token) - - -async def get_authenticated_user(api_token: str = Depends(token_scheme)) -> User: - user, token = await __get_authenticated_user(api_token) - return user - - @sync_to_async def __get_login_user(username: str) -> User: kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 74730ff..a60c7f0 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -5,8 +5,7 @@ from django.contrib.auth import get_user_model from django.core import exceptions as django_exceptions from django.core.files.base import ContentFile from django.db import transaction -from django.db.models import Q -from django.db.models import QuerySet +from django.db.models import Q, QuerySet from fastapi import APIRouter, Depends, status from django_etebase import models @@ -25,12 +24,11 @@ from .utils import ( PERMISSIONS_READ, PERMISSIONS_READWRITE, ) +from .dependencies import get_collection_queryset, get_item_queryset, get_collection User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) -default_queryset: QuerySet = models.Collection.objects.all() -default_item_queryset: QuerySet = models.CollectionItem.objects.all() class ListMulti(BaseModel): @@ -203,21 +201,6 @@ def collection_list_common( return ret -def get_collection_queryset(user: User = Depends(get_authenticated_user)) -> QuerySet: - return default_queryset.filter(members__user=user) - - -def get_collection(collection_uid: str, queryset: QuerySet = Depends(get_collection_queryset)) -> models.Collection: - return get_object_or_404(queryset, uid=collection_uid) - - -def get_item_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: - # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') - queryset = default_item_queryset.filter(collection__pk=collection.pk, revisions__current=True) - - return queryset - - # permissions diff --git a/etebase_fastapi/dependencies.py b/etebase_fastapi/dependencies.py new file mode 100644 index 0000000..ddb9b3b --- /dev/null +++ b/etebase_fastapi/dependencies.py @@ -0,0 +1,82 @@ +import dataclasses + +from fastapi import Depends +from fastapi.security import APIKeyHeader + +from django.contrib.auth import get_user_model +from django.utils import timezone +from django.db.models import QuerySet + +from django_etebase import models +from django_etebase.token_auth.models import AuthToken, get_default_expiry +from .exceptions import AuthenticationFailed +from .utils import get_object_or_404 + + +User = get_user_model() +token_scheme = APIKeyHeader(name="Authorization") +AUTO_REFRESH = True +MIN_REFRESH_INTERVAL = 60 + + +@dataclasses.dataclass(frozen=True) +class AuthData: + user: User + token: AuthToken + + +def __renew_token(auth_token: AuthToken): + current_expiry = auth_token.expiry + new_expiry = get_default_expiry() + # Throttle refreshing of token to avoid db writes + delta = (new_expiry - current_expiry).total_seconds() + if delta > MIN_REFRESH_INTERVAL: + auth_token.expiry = new_expiry + auth_token.save(update_fields=("expiry",)) + + +def __get_authenticated_user(api_token: str): + api_token = api_token.split()[1] + try: + token: AuthToken = AuthToken.objects.select_related("user").get(key=api_token) + except AuthToken.DoesNotExist: + raise AuthenticationFailed(detail="Invalid token.") + if not token.user.is_active: + raise AuthenticationFailed(detail="User inactive or deleted.") + + if token.expiry is not None: + if token.expiry < timezone.now(): + token.delete() + raise AuthenticationFailed(detail="Invalid token.") + + if AUTO_REFRESH: + __renew_token(token) + + return token.user, token + + +def get_auth_data(api_token: str = Depends(token_scheme)) -> AuthData: + user, token = __get_authenticated_user(api_token) + return AuthData(user, token) + + +def get_authenticated_user(api_token: str = Depends(token_scheme)) -> User: + user, _ = __get_authenticated_user(api_token) + return user + + +def get_collection_queryset(user: User = Depends(get_authenticated_user)) -> QuerySet: + default_queryset: QuerySet = models.Collection.objects.all() + return default_queryset.filter(members__user=user) + + +def get_collection(collection_uid: str, queryset: QuerySet = Depends(get_collection_queryset)) -> models.Collection: + return get_object_or_404(queryset, uid=collection_uid) + + +def get_item_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: + default_item_queryset: QuerySet = models.CollectionItem.objects.all() + # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') + queryset = default_item_queryset.filter(collection__pk=collection.pk, revisions__current=True) + + return queryset From 586b015eb78a01d40a13c9acad81c274b5be6380 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 16:23:01 +0200 Subject: [PATCH 440/511] Login: also return username and email upon login. --- etebase_fastapi/authentication.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index eb54f68..a211e9b 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -46,12 +46,19 @@ class LoginResponse(BaseModel): class UserOut(BaseModel): + username: str + email: str pubkey: bytes encryptedContent: bytes @classmethod def from_orm(cls: t.Type["UserOut"], obj: User) -> "UserOut": - return cls(pubkey=bytes(obj.userinfo.pubkey), encryptedContent=bytes(obj.userinfo.encryptedContent)) + return cls( + username=obj.username, + email=obj.email, + pubkey=bytes(obj.userinfo.pubkey), + encryptedContent=bytes(obj.userinfo.encryptedContent), + ) class LoginOut(BaseModel): From 151bec0d9e683a0c8da93796a7468c999c6aecd0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 16:44:13 +0200 Subject: [PATCH 441/511] Fix type error. --- etebase_fastapi/authentication.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index a211e9b..559a60a 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -261,6 +261,6 @@ def signup_save(data: SignupIn, request: Request) -> User: @authentication_router.post("/signup/", response_model=LoginOut, status_code=status.HTTP_201_CREATED) def signup(data: SignupIn, request: Request): user = signup_save(data, request) - data = LoginOut.from_orm(user) + ret = LoginOut.from_orm(user) user_signed_up.send(sender=user.__class__, request=None, user=user) - return data + return ret From fa0979dce17c779321325c6ef173addb47af52f0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 16:57:09 +0200 Subject: [PATCH 442/511] Test reset: clean reset function. --- etebase_fastapi/test_reset_view.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py index f21fd84..3075290 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/test_reset_view.py @@ -2,31 +2,33 @@ from django.conf import settings from django.contrib.auth import get_user_model from django.db import transaction from django.shortcuts import get_object_or_404 -from fastapi import APIRouter, Request, Response, status +from fastapi import APIRouter, Request, status from django_etebase.utils import get_user_queryset, CallbackContext from etebase_fastapi.authentication import SignupIn, signup_save from etebase_fastapi.msgpack import MsgpackRoute +from etebase_fastapi.exceptions import HttpError test_reset_view_router = APIRouter(route_class=MsgpackRoute, tags=["test helpers"]) User = get_user_model() -@test_reset_view_router.post("/reset/") +@test_reset_view_router.post("/reset/", status_code=status.HTTP_204_NO_CONTENT) def reset(data: SignupIn, request: Request): # Only run when in DEBUG mode! It's only used for tests if not settings.DEBUG: - return Response("Only allowed in debug mode.", status_code=status.HTTP_400_BAD_REQUEST) + raise HttpError(code="generic", detail="Only allowed in debug mode.") with transaction.atomic(): user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) user = get_object_or_404(user_queryset, username=data.user.username) # Only allow test users for extra safety if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): - return Response("Endpoint not allowed for user.", status_code=status.HTTP_400_BAD_REQUEST) + raise HttpError(code="generic", detail="Endpoint not allowed for user.") if hasattr(user, "userinfo"): user.userinfo.delete() + signup_save(data, request) # Delete all of the journal data for this user for a clear test env user.collection_set.all().delete() @@ -34,5 +36,3 @@ def reset(data: SignupIn, request: Request): user.incoming_invitations.all().delete() # FIXME: also delete chunk files!!! - - return Response(status_code=status.HTTP_204_NO_CONTENT) From 10ff303b754f8147acdc4f1740d27bf3dbefa915 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 17:09:20 +0200 Subject: [PATCH 443/511] Fix formatting. --- etebase_fastapi/main.py | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index 706081a..a55e2fa 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -12,6 +12,7 @@ from .member import member_router from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse + def create_application(prefix=""): app = FastAPI() VERSION = "v1" @@ -21,8 +22,12 @@ def create_application(prefix=""): app.include_router(collection_router, prefix=f"{BASE_PATH}/collection", tags=["collection"]) app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["item"]) app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["member"]) - app.include_router(invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"]) - app.include_router(invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"]) + app.include_router( + invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"] + ) + app.include_router( + invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"] + ) if settings.DEBUG: from etebase_fastapi.test_reset_view import test_reset_view_router @@ -30,12 +35,16 @@ def create_application(prefix=""): app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") app.add_middleware( - CORSMiddleware, allow_origin_regex="https?://.*", allow_credentials=True, allow_methods=["*"], allow_headers=["*"] + CORSMiddleware, + allow_origin_regex="https?://.*", + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], ) app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) - @app.exception_handler(CustomHttpException) async def custom_exception_handler(request: Request, exc: CustomHttpException): return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) + return app From 3d438b9591955e948c71c3737a534392b5b9a0ef Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 17:39:51 +0200 Subject: [PATCH 444/511] Cleanup validation errors. --- etebase_fastapi/collection.py | 41 +++++++++++++++++++++++++++-------- etebase_fastapi/exceptions.py | 19 ++++++++++++++++ 2 files changed, 51 insertions(+), 9 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index a60c7f0..6c7d9a4 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -10,7 +10,7 @@ from fastapi import APIRouter, Depends, status from django_etebase import models from .authentication import get_authenticated_user -from .exceptions import HttpError, transform_validation_error, PermissionDenied +from .exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError from .msgpack import MsgpackRoute from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken from .utils import ( @@ -151,10 +151,11 @@ class ItemDepIn(BaseModel): item = models.CollectionItem.objects.get(uid=self.uid) etag = self.etag if item.etag != etag: - raise HttpError( + raise ValidationError( "wrong_etag", "Wrong etag. Expected {} got {}".format(item.etag, etag), status_code=status.HTTP_409_CONFLICT, + field=self.uid, ) @@ -164,8 +165,19 @@ class ItemBatchIn(BaseModel): def validate_db(self): if self.deps is not None: + errors: t.List[HttpError] = [] for dep in self.deps: - dep.validate_db() + try: + dep.validate_db() + except ValidationError as e: + errors.append(e) + if len(errors) > 0: + raise ValidationError( + code="dep_failed", + detail="Dependencies failed to validate", + errors=errors, + status_code=status.HTTP_409_CONFLICT, + ) @sync_to_async @@ -293,7 +305,7 @@ def _create(data: CollectionIn, user: User): try: instance.validate_unique() except django_exceptions.ValidationError: - raise HttpError( + raise ValidationError( "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT ) instance.save() @@ -353,10 +365,11 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val return instance if validate_etag and cur_etag != etag: - raise HttpError( + raise ValidationError( "wrong_etag", "Wrong etag. Expected {} got {}".format(cur_etag, etag), status_code=status.HTTP_409_CONFLICT, + field=uid, ) if not created: @@ -426,12 +439,22 @@ def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid if stoken is not None and stoken != collection_object.stoken: raise HttpError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) - # XXX-TOM: make sure we return compatible errors data.validate_db() - for item in data.items: - item_create(item, collection_object, validate_etag) - return None + errors: t.List[HttpError] = [] + for item in data.items: + try: + item_create(item, collection_object, validate_etag) + except ValidationError as e: + errors.append(e) + + if len(errors) > 0: + raise ValidationError( + code="item_failed", + detail="Items failed to validate", + errors=errors, + status_code=status.HTTP_409_CONFLICT, + ) @item_router.get( diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index 2c1757c..72a3faf 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -9,12 +9,18 @@ class HttpErrorField(BaseModel): code: str detail: str + class Config: + orm_mode = True + class HttpErrorOut(BaseModel): code: str detail: str errors: t.Optional[t.List[HttpErrorField]] + class Config: + orm_mode = True + class CustomHttpException(Exception): def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): @@ -73,6 +79,19 @@ class HttpError(CustomHttpException): return HttpErrorOut(code=self.code, errors=self.errors, detail=self.detail).dict() +class ValidationError(HttpError): + def __init__( + self, + code: str, + detail: str, + status_code: int = status.HTTP_400_BAD_REQUEST, + errors: t.Optional[t.List["HttpError"]] = None, + field: t.Optional[str] = None, + ): + self.field = field + super().__init__(code=code, detail=detail, errors=errors, status_code=status_code) + + def flatten_errors(field_name, errors) -> t.List[HttpError]: ret = [] if isinstance(errors, dict): From f7858a20b7c7165908240aaa2ada4278b17260db Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 17:46:20 +0200 Subject: [PATCH 445/511] Fix user creation. --- django_etebase/utils.py | 2 +- etebase_fastapi/authentication.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/django_etebase/utils.py b/django_etebase/utils.py index e46cbd0..4d36a94 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -28,7 +28,7 @@ def get_user_queryset(queryset, context: CallbackContext): def create_user(context: CallbackContext, *args, **kwargs): custom_func = app_settings.CREATE_USER_FUNC if custom_func is not None: - return custom_func(*args, **kwargs) + return custom_func(context, *args, **kwargs) return User.objects.create_user(*args, **kwargs) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 559a60a..fe522f7 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -181,7 +181,7 @@ async def is_etebase(): @authentication_router.post("/login_challenge/", response_model=LoginChallengeOut) -async def login_challenge(user: User = Depends(get_login_user)): +def login_challenge(user: User = Depends(get_login_user)): salt = bytes(user.userinfo.salt) enc_key = get_encryption_key(salt) box = nacl.secret.SecretBox(enc_key) @@ -238,10 +238,10 @@ def signup_save(data: SignupIn, request: Request) -> User: # Create the user and save the casing the user chose as the first name try: instance = create_user( + CallbackContext(request.path_params), **user_data.dict(), password=None, first_name=user_data.username, - context=CallbackContext(request.path_params), ) instance.full_clean() except HttpError as e: From dcf81aa9ceeca7b56f84f47028ef05eda2980406 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 18:17:41 +0200 Subject: [PATCH 446/511] Fix prefetch medium. --- etebase_fastapi/collection.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 6c7d9a4..07780ae 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -55,7 +55,7 @@ class CollectionItemRevisionInOut(BaseModel): with open(chunk_obj.chunkFile.path, "rb") as f: chunks.append((chunk_obj.uid, f.read())) else: - chunks.append((chunk_obj.uid,)) + chunks.append((chunk_obj.uid, None)) return cls(uid=obj.uid, meta=bytes(obj.meta), deleted=obj.deleted, chunks=chunks) From 53b22602b28bf2582284495d736b2b1672d35a9b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 18:17:57 +0200 Subject: [PATCH 447/511] Implement chunk_update. --- etebase_fastapi/collection.py | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 07780ae..b3f8c5f 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -4,9 +4,9 @@ from asgiref.sync import sync_to_async from django.contrib.auth import get_user_model from django.core import exceptions as django_exceptions from django.core.files.base import ContentFile -from django.db import transaction +from django.db import transaction, IntegrityError from django.db.models import Q, QuerySet -from fastapi import APIRouter, Depends, status +from fastapi import APIRouter, Depends, status, Request from django_etebase import models from .authentication import get_authenticated_user @@ -544,28 +544,27 @@ def item_batch( # Chunks +@sync_to_async +def chunk_save(chunk_uid: str, collection: models.Collection, content_file: ContentFile): + chunk_obj = models.CollectionItemChunk(uid=chunk_uid, collection=collection) + chunk_obj.chunkFile.save("IGNORED", content_file) + chunk_obj.save() + return chunk_obj + + @item_router.put( "/item/{item_uid}/chunk/{chunk_uid}/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE], status_code=status.HTTP_201_CREATED, ) -def chunk_update( - limit: int = 50, - iterator: t.Optional[str] = None, - prefetch: Prefetch = PrefetchQuery, - user: User = Depends(get_authenticated_user), +async def chunk_update( + request: Request, + chunk_uid: str, collection: models.Collection = Depends(get_collection), ): # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) - - data = { - "uid": chunk_uid, - "chunkFile": request.data["file"], - } - - serializer = self.get_serializer_class()(data=data) - serializer.is_valid(raise_exception=True) + content_file = ContentFile(await request.body()) try: - serializer.save(collection=col) + await chunk_save(chunk_uid, collection, content_file) except IntegrityError: - return Response({"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT) + raise HttpError("chunk_exists", "Chunk already exists.", status_code=status.HTTP_409_CONFLICT) From c7f09d3fef2935b81ac3a9232a6b88215b73300d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 18:25:06 +0200 Subject: [PATCH 448/511] implement chunk_download. --- etebase_fastapi/collection.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index b3f8c5f..c5b0801 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -7,6 +7,7 @@ from django.core.files.base import ContentFile from django.db import transaction, IntegrityError from django.db.models import Q, QuerySet from fastapi import APIRouter, Depends, status, Request +from fastapi.responses import FileResponse from django_etebase import models from .authentication import get_authenticated_user @@ -568,3 +569,17 @@ async def chunk_update( await chunk_save(chunk_uid, collection, content_file) except IntegrityError: raise HttpError("chunk_exists", "Chunk already exists.", status_code=status.HTTP_409_CONFLICT) + + +@item_router.get( + "/item/{item_uid}/chunk/{chunk_uid}/download/", + dependencies=PERMISSIONS_READ, +) +def chunk_download( + chunk_uid: str, + collection: models.Collection = Depends(get_collection), +): + chunk = get_object_or_404(collection.chunks, uid=chunk_uid) + + filename = chunk.chunkFile.path + return FileResponse(filename, media_type="application/octet-stream") From f0a8689712601ca2ac5b98b5c3c4aad4f6d58808 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 18:44:55 +0200 Subject: [PATCH 449/511] Implement sendfile. --- .../sendfile/backends/development.py | 17 ------ django_etebase/sendfile/backends/mod_wsgi.py | 17 ------ django_etebase/sendfile/backends/nginx.py | 12 ---- django_etebase/sendfile/backends/simple.py | 60 ------------------- django_etebase/sendfile/backends/xsendfile.py | 9 --- etebase_fastapi/collection.py | 4 +- .../sendfile/LICENSE | 0 .../sendfile/README.md | 0 .../sendfile/__init__.py | 0 .../sendfile/backends/__init__.py | 0 etebase_fastapi/sendfile/backends/mod_wsgi.py | 9 +++ etebase_fastapi/sendfile/backends/nginx.py | 9 +++ etebase_fastapi/sendfile/backends/simple.py | 12 ++++ .../sendfile/backends/xsendfile.py | 6 ++ .../sendfile/utils.py | 15 ++--- 15 files changed, 46 insertions(+), 124 deletions(-) delete mode 100644 django_etebase/sendfile/backends/development.py delete mode 100644 django_etebase/sendfile/backends/mod_wsgi.py delete mode 100644 django_etebase/sendfile/backends/nginx.py delete mode 100644 django_etebase/sendfile/backends/simple.py delete mode 100644 django_etebase/sendfile/backends/xsendfile.py rename {django_etebase => etebase_fastapi}/sendfile/LICENSE (100%) rename {django_etebase => etebase_fastapi}/sendfile/README.md (100%) rename {django_etebase => etebase_fastapi}/sendfile/__init__.py (100%) rename {django_etebase => etebase_fastapi}/sendfile/backends/__init__.py (100%) create mode 100644 etebase_fastapi/sendfile/backends/mod_wsgi.py create mode 100644 etebase_fastapi/sendfile/backends/nginx.py create mode 100644 etebase_fastapi/sendfile/backends/simple.py create mode 100644 etebase_fastapi/sendfile/backends/xsendfile.py rename {django_etebase => etebase_fastapi}/sendfile/utils.py (81%) diff --git a/django_etebase/sendfile/backends/development.py b/django_etebase/sendfile/backends/development.py deleted file mode 100644 index d321932..0000000 --- a/django_etebase/sendfile/backends/development.py +++ /dev/null @@ -1,17 +0,0 @@ -import os.path - -from django.views.static import serve - - -def sendfile(request, filename, **kwargs): - """ - Send file using Django dev static file server. - - .. warning:: - - Do not use in production. This is only to be used when developing and - is provided for convenience only - """ - dirname = os.path.dirname(filename) - basename = os.path.basename(filename) - return serve(request, basename, dirname) diff --git a/django_etebase/sendfile/backends/mod_wsgi.py b/django_etebase/sendfile/backends/mod_wsgi.py deleted file mode 100644 index 07ba3f1..0000000 --- a/django_etebase/sendfile/backends/mod_wsgi.py +++ /dev/null @@ -1,17 +0,0 @@ -from __future__ import absolute_import - -from django.http import HttpResponse - -from ..utils import _convert_file_to_url - - -def sendfile(request, filename, **kwargs): - response = HttpResponse() - response['Location'] = _convert_file_to_url(filename) - # need to destroy get_host() to stop django - # rewriting our location to include http, so that - # mod_wsgi is able to do the internal redirect - request.get_host = lambda: '' - request.build_absolute_uri = lambda location: location - - return response diff --git a/django_etebase/sendfile/backends/nginx.py b/django_etebase/sendfile/backends/nginx.py deleted file mode 100644 index 8764309..0000000 --- a/django_etebase/sendfile/backends/nginx.py +++ /dev/null @@ -1,12 +0,0 @@ -from __future__ import absolute_import - -from django.http import HttpResponse - -from ..utils import _convert_file_to_url - - -def sendfile(request, filename, **kwargs): - response = HttpResponse() - response['X-Accel-Redirect'] = _convert_file_to_url(filename) - - return response diff --git a/django_etebase/sendfile/backends/simple.py b/django_etebase/sendfile/backends/simple.py deleted file mode 100644 index 0549b20..0000000 --- a/django_etebase/sendfile/backends/simple.py +++ /dev/null @@ -1,60 +0,0 @@ -from email.utils import mktime_tz, parsedate_tz -import re - -from django.core.files.base import File -from django.http import HttpResponse, HttpResponseNotModified -from django.utils.http import http_date - - -def sendfile(request, filepath, **kwargs): - '''Use the SENDFILE_ROOT value composed with the path arrived as argument - to build an absolute path with which resolve and return the file contents. - - If the path points to a file out of the root directory (should cover both - situations with '..' and symlinks) then a 404 is raised. - ''' - statobj = filepath.stat() - - # Respect the If-Modified-Since header. - if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), - statobj.st_mtime, statobj.st_size): - return HttpResponseNotModified() - - with File(filepath.open('rb')) as f: - response = HttpResponse(f.chunks()) - - response["Last-Modified"] = http_date(statobj.st_mtime) - return response - - -def was_modified_since(header=None, mtime=0, size=0): - """ - Was something modified since the user last downloaded it? - - header - This is the value of the If-Modified-Since header. If this is None, - I'll just return True. - - mtime - This is the modification time of the item we're talking about. - - size - This is the size of the item we're talking about. - """ - try: - if header is None: - raise ValueError - matches = re.match(r"^([^;]+)(; length=([0-9]+))?$", header, - re.IGNORECASE) - header_date = parsedate_tz(matches.group(1)) - if header_date is None: - raise ValueError - header_mtime = mktime_tz(header_date) - header_len = matches.group(3) - if header_len and int(header_len) != size: - raise ValueError - if mtime > header_mtime: - raise ValueError - except (AttributeError, ValueError, OverflowError): - return True - return False diff --git a/django_etebase/sendfile/backends/xsendfile.py b/django_etebase/sendfile/backends/xsendfile.py deleted file mode 100644 index 74993ee..0000000 --- a/django_etebase/sendfile/backends/xsendfile.py +++ /dev/null @@ -1,9 +0,0 @@ -from django.http import HttpResponse - - -def sendfile(request, filename, **kwargs): - filename = str(filename) - response = HttpResponse() - response['X-Sendfile'] = filename - - return response diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index c5b0801..3b672cc 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -7,7 +7,6 @@ from django.core.files.base import ContentFile from django.db import transaction, IntegrityError from django.db.models import Q, QuerySet from fastapi import APIRouter, Depends, status, Request -from fastapi.responses import FileResponse from django_etebase import models from .authentication import get_authenticated_user @@ -26,6 +25,7 @@ from .utils import ( PERMISSIONS_READWRITE, ) from .dependencies import get_collection_queryset, get_item_queryset, get_collection +from .sendfile import sendfile User = get_user_model() collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -582,4 +582,4 @@ def chunk_download( chunk = get_object_or_404(collection.chunks, uid=chunk_uid) filename = chunk.chunkFile.path - return FileResponse(filename, media_type="application/octet-stream") + return sendfile(filename) diff --git a/django_etebase/sendfile/LICENSE b/etebase_fastapi/sendfile/LICENSE similarity index 100% rename from django_etebase/sendfile/LICENSE rename to etebase_fastapi/sendfile/LICENSE diff --git a/django_etebase/sendfile/README.md b/etebase_fastapi/sendfile/README.md similarity index 100% rename from django_etebase/sendfile/README.md rename to etebase_fastapi/sendfile/README.md diff --git a/django_etebase/sendfile/__init__.py b/etebase_fastapi/sendfile/__init__.py similarity index 100% rename from django_etebase/sendfile/__init__.py rename to etebase_fastapi/sendfile/__init__.py diff --git a/django_etebase/sendfile/backends/__init__.py b/etebase_fastapi/sendfile/backends/__init__.py similarity index 100% rename from django_etebase/sendfile/backends/__init__.py rename to etebase_fastapi/sendfile/backends/__init__.py diff --git a/etebase_fastapi/sendfile/backends/mod_wsgi.py b/etebase_fastapi/sendfile/backends/mod_wsgi.py new file mode 100644 index 0000000..b8fc6c0 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/mod_wsgi.py @@ -0,0 +1,9 @@ +from __future__ import absolute_import + +from fastapi import Response + +from ..utils import _convert_file_to_url + + +def sendfile(filename, **kwargs): + return Response(headers={"Location": _convert_file_to_url(filename)}) diff --git a/etebase_fastapi/sendfile/backends/nginx.py b/etebase_fastapi/sendfile/backends/nginx.py new file mode 100644 index 0000000..b22e0d0 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/nginx.py @@ -0,0 +1,9 @@ +from __future__ import absolute_import + +from fastapi import Response + +from ..utils import _convert_file_to_url + + +def sendfile(filename, **kwargs): + return Response(headers={"X-Accel-Redirect": _convert_file_to_url(filename)}) diff --git a/etebase_fastapi/sendfile/backends/simple.py b/etebase_fastapi/sendfile/backends/simple.py new file mode 100644 index 0000000..f3a3548 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/simple.py @@ -0,0 +1,12 @@ +from fastapi.responses import FileResponse + + +def sendfile(filename, mimetype, **kwargs): + """Use the SENDFILE_ROOT value composed with the path arrived as argument + to build an absolute path with which resolve and return the file contents. + + If the path points to a file out of the root directory (should cover both + situations with '..' and symlinks) then a 404 is raised. + """ + + return FileResponse(filename, media_type=mimetype) diff --git a/etebase_fastapi/sendfile/backends/xsendfile.py b/etebase_fastapi/sendfile/backends/xsendfile.py new file mode 100644 index 0000000..530f6a1 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/xsendfile.py @@ -0,0 +1,6 @@ +from fastapi import Response + + +def sendfile(filename, **kwargs): + filename = str(filename) + return Response(headers={"X-Sendfile": filename}) diff --git a/django_etebase/sendfile/utils.py b/etebase_fastapi/sendfile/utils.py similarity index 81% rename from django_etebase/sendfile/utils.py rename to etebase_fastapi/sendfile/utils.py index 97c06d7..7c8b1f2 100644 --- a/django_etebase/sendfile/utils.py +++ b/etebase_fastapi/sendfile/utils.py @@ -4,9 +4,11 @@ from pathlib import Path, PurePath from urllib.parse import quote import logging +from fastapi import status +from ..exceptions import HttpError + from django.conf import settings from django.core.exceptions import ImproperlyConfigured -from django.http import Http404 logger = logging.getLogger(__name__) @@ -54,12 +56,12 @@ def _sanitize_path(filepath): try: filepath_abs.relative_to(path_root) except ValueError: - raise Http404("{} wrt {} is impossible".format(filepath_abs, path_root)) + raise HttpError("generic", "{} wrt {} is impossible".format(filepath_abs, path_root), status_code=status.HTTP_404_NOT_FOUND) return filepath_abs -def sendfile(request, filename, mimetype="application/octet-stream", encoding=None): +def sendfile(filename, mimetype="application/octet-stream", encoding=None): """ Create a response to send file using backend configured in ``SENDFILE_BACKEND`` @@ -75,11 +77,10 @@ def sendfile(request, filename, mimetype="application/octet-stream", encoding=No _sendfile = _get_sendfile() if not filepath_obj.exists(): - raise Http404('"%s" does not exist' % filepath_obj) + raise HttpError("does_not_exist", '"%s" does not exist' % filepath_obj, status_code=status.HTTP_404_NOT_FOUND) - response = _sendfile(request, filepath_obj, mimetype=mimetype) + response = _sendfile(filepath_obj, mimetype=mimetype) - response["Content-length"] = filepath_obj.stat().st_size - response["Content-Type"] = mimetype + response.headers["Content-Type"] = mimetype return response From 7714148807a183ef4eb820232d4014662441372a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 28 Dec 2020 18:49:05 +0200 Subject: [PATCH 450/511] Use ValidationError when appropriate. --- etebase_fastapi/collection.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 3b672cc..3e4d3e8 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -285,7 +285,7 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) chunk_obj.save() else: - raise HttpError("chunk_no_content", "Tried to create a new chunk without content") + raise ValidationError("chunk_no_content", "Tried to create a new chunk without content") chunks_objs.append(chunk_obj) @@ -301,7 +301,7 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle def _create(data: CollectionIn, user: User): with transaction.atomic(): if data.item.etag is not None: - raise HttpError("bad_etag", "etag is not null") + raise ValidationError("bad_etag", "etag is not null") instance = models.Collection(uid=data.item.uid, owner=user) try: instance.validate_unique() From a8b97e60d407659f363d3486cbde1f8ad5d7bdd7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 09:46:20 +0200 Subject: [PATCH 451/511] Docs: improve metadata. --- etebase_fastapi/main.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index a55e2fa..69303bf 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -14,7 +14,15 @@ from .msgpack import MsgpackResponse def create_application(prefix=""): - app = FastAPI() + app = FastAPI( + title="Etebase", + description="The Etebase server API documentation", + externalDocs={ + "url": "https://docs.etebase.com", + "description": "Docs about the API specifications and clients.", + } + # FIXME: version="2.5.0", + ) VERSION = "v1" BASE_PATH = f"{prefix}/api/{VERSION}" COLLECTION_UID_MARKER = "{collection_uid}" From f67730f42d2a78acadf28db253bf76c0f3630ba2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 10:12:36 +0200 Subject: [PATCH 452/511] Support passing custom middlewares. --- etebase_fastapi/main.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index 69303bf..534798e 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -13,7 +13,7 @@ from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse -def create_application(prefix=""): +def create_application(prefix="", middlewares=[]): app = FastAPI( title="Etebase", description="The Etebase server API documentation", @@ -51,6 +51,9 @@ def create_application(prefix=""): ) app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) + for middleware in middlewares: + app.add_middleware(middleware) + @app.exception_handler(CustomHttpException) async def custom_exception_handler(request: Request, exc: CustomHttpException): return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) From e13f26ec56f03321ad5f8471f4a5a33f4a6f0be0 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 10:27:35 +0200 Subject: [PATCH 453/511] Fix handling of legacy accounts that don't have collection type. --- etebase_fastapi/collection.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 3e4d3e8..5c6e6b6 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -88,7 +88,8 @@ class CollectionItemIn(CollectionItemCommon): class CollectionCommon(BaseModel): - collectionType: bytes + # FIXME: remove optional once we finish collection-type-migration + collectionType: t.Optional[bytes] collectionKey: bytes From 794b5f398347985d7a354acc16e8fcd536600358 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 13:22:36 +0200 Subject: [PATCH 454/511] Fix many type errors. --- django_etebase/token_auth/models.py | 4 +-- django_etebase/utils.py | 6 ++-- etebase_fastapi/authentication.py | 27 ++++++++--------- etebase_fastapi/collection.py | 45 +++++++++++++++-------------- etebase_fastapi/dependencies.py | 10 +++---- etebase_fastapi/invitation.py | 15 +++++----- etebase_fastapi/member.py | 8 +++-- etebase_fastapi/msgpack.py | 6 ++-- etebase_fastapi/test_reset_view.py | 4 +-- etebase_fastapi/utils.py | 6 ++-- myauth/forms.py | 4 +-- myauth/models.py | 16 +++++++++- 12 files changed, 87 insertions(+), 64 deletions(-) diff --git a/django_etebase/token_auth/models.py b/django_etebase/token_auth/models.py index ac1efff..dd5ae87 100644 --- a/django_etebase/token_auth/models.py +++ b/django_etebase/token_auth/models.py @@ -1,9 +1,9 @@ -from django.contrib.auth import get_user_model from django.db import models from django.utils import timezone from django.utils.crypto import get_random_string +from myauth.models import get_typed_user_model -User = get_user_model() +User = get_typed_user_model() def generate_key(): diff --git a/django_etebase/utils.py b/django_etebase/utils.py index 4d36a94..d812ae3 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -1,13 +1,13 @@ import typing as t from dataclasses import dataclass -from django.contrib.auth import get_user_model from django.core.exceptions import PermissionDenied +from myauth.models import UserType, get_typed_user_model from . import app_settings -User = get_user_model() +User = get_typed_user_model() @dataclass @@ -15,7 +15,7 @@ class CallbackContext: """Class for passing extra context to callbacks""" url_kwargs: t.Dict[str, t.Any] - user: t.Optional[User] = None + user: t.Optional[UserType] = None def get_user_queryset(queryset, context: CallbackContext): diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index fe522f7..064d2da 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -9,7 +9,7 @@ import nacl.secret import nacl.signing from asgiref.sync import sync_to_async from django.conf import settings -from django.contrib.auth import get_user_model, user_logged_out, user_logged_in +from django.contrib.auth import user_logged_out, user_logged_in from django.core import exceptions as django_exceptions from django.db import transaction from fastapi import APIRouter, Depends, status, Request @@ -19,12 +19,13 @@ from django_etebase.token_auth.models import AuthToken from django_etebase.models import UserInfo from django_etebase.signals import user_signed_up from django_etebase.utils import create_user, get_user_queryset, CallbackContext +from myauth.models import UserType, get_typed_user_model from .exceptions import AuthenticationFailed, transform_validation_error, HttpError from .msgpack import MsgpackRoute from .utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode from .dependencies import AuthData, get_auth_data, get_authenticated_user -User = get_user_model() +User = get_typed_user_model() authentication_router = APIRouter(route_class=MsgpackRoute) @@ -52,7 +53,7 @@ class UserOut(BaseModel): encryptedContent: bytes @classmethod - def from_orm(cls: t.Type["UserOut"], obj: User) -> "UserOut": + def from_orm(cls: t.Type["UserOut"], obj: UserType) -> "UserOut": return cls( username=obj.username, email=obj.email, @@ -66,7 +67,7 @@ class LoginOut(BaseModel): user: UserOut @classmethod - def from_orm(cls: t.Type["LoginOut"], obj: User) -> "LoginOut": + def from_orm(cls: t.Type["LoginOut"], obj: UserType) -> "LoginOut": token = AuthToken.objects.create(user=obj).key user = UserOut.from_orm(obj) return cls(token=token, user=user) @@ -111,7 +112,7 @@ class SignupIn(BaseModel): @sync_to_async -def __get_login_user(username: str) -> User: +def __get_login_user(username: str) -> UserType: kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} try: user = User.objects.get(**kwargs) @@ -122,7 +123,7 @@ def __get_login_user(username: str) -> User: raise AuthenticationFailed(code="user_not_found", detail="User not found") -async def get_login_user(challenge: LoginChallengeIn) -> User: +async def get_login_user(challenge: LoginChallengeIn) -> UserType: user = await __get_login_user(challenge.username) return user @@ -138,7 +139,7 @@ def get_encryption_key(salt): ) -def save_changed_password(data: ChangePassword, user: User): +def save_changed_password(data: ChangePassword, user: UserType): response_data = data.response_data user_info: UserInfo = user.userinfo user_info.loginPubkey = response_data.loginPubkey @@ -150,7 +151,7 @@ def save_changed_password(data: ChangePassword, user: User): def validate_login_request( validated_data: LoginResponse, challenge_sent_to_user: Authentication, - user: User, + user: UserType, expected_action: str, host_from_request: str, ): @@ -159,7 +160,7 @@ def validate_login_request( challenge_data = msgpack_decode(box.decrypt(validated_data.challenge)) now = int(datetime.now().timestamp()) if validated_data.action != expected_action: - raise HttpError("wrong_action", f'Expected "{challenge_sent_to_user.response}" but got something else') + raise HttpError("wrong_action", f'Expected "{expected_action}" but got something else') elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: raise HttpError("challenge_expired", "Login challenge has expired") elif challenge_data["userId"] != user.id: @@ -181,7 +182,7 @@ async def is_etebase(): @authentication_router.post("/login_challenge/", response_model=LoginChallengeOut) -def login_challenge(user: User = Depends(get_login_user)): +def login_challenge(user: UserType = Depends(get_login_user)): salt = bytes(user.userinfo.salt) enc_key = get_encryption_key(salt) box = nacl.secret.SecretBox(enc_key) @@ -210,14 +211,14 @@ def logout(auth_data: AuthData = Depends(get_auth_data)): @authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) -async def change_password(data: ChangePassword, request: Request, user: User = Depends(get_authenticated_user)): +async def change_password(data: ChangePassword, request: Request, user: UserType = Depends(get_authenticated_user)): host = request.headers.get("Host") await validate_login_request(data.response_data, data, user, "changePassword", host) await sync_to_async(save_changed_password)(data, user) @authentication_router.post("/dashboard_url/", responses=permission_responses) -def dashboard_url(request: Request, user: User = Depends(get_authenticated_user)): +def dashboard_url(request: Request, user: UserType = Depends(get_authenticated_user)): get_dashboard_url = app_settings.DASHBOARD_URL_FUNC if get_dashboard_url is None: raise HttpError("not_supported", "This server doesn't have a user dashboard.") @@ -228,7 +229,7 @@ def dashboard_url(request: Request, user: User = Depends(get_authenticated_user) return ret -def signup_save(data: SignupIn, request: Request) -> User: +def signup_save(data: SignupIn, request: Request) -> UserType: user_data = data.user with transaction.atomic(): try: diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 5c6e6b6..9e25b38 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -1,7 +1,6 @@ import typing as t from asgiref.sync import sync_to_async -from django.contrib.auth import get_user_model from django.core import exceptions as django_exceptions from django.core.files.base import ContentFile from django.db import transaction, IntegrityError @@ -9,6 +8,7 @@ from django.db.models import Q, QuerySet from fastapi import APIRouter, Depends, status, Request from django_etebase import models +from myauth.models import UserType, get_typed_user_model from .authentication import get_authenticated_user from .exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError from .msgpack import MsgpackRoute @@ -27,7 +27,7 @@ from .utils import ( from .dependencies import get_collection_queryset, get_item_queryset, get_collection from .sendfile import sendfile -User = get_user_model() +User = get_typed_user_model collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -36,11 +36,14 @@ class ListMulti(BaseModel): collectionTypes: t.List[bytes] +ChunkType = t.Tuple[str, t.Optional[bytes]] + + class CollectionItemRevisionInOut(BaseModel): uid: str meta: bytes deleted: bool - chunks: t.List[t.Tuple[str, t.Optional[bytes]]] + chunks: t.List[ChunkType] class Config: orm_mode = True @@ -49,7 +52,7 @@ class CollectionItemRevisionInOut(BaseModel): def from_orm_context( cls: t.Type["CollectionItemRevisionInOut"], obj: models.CollectionItemRevision, context: Context ) -> "CollectionItemRevisionInOut": - chunks = [] + chunks: t.List[ChunkType] = [] for chunk_relation in obj.chunks_relation.all(): chunk_obj = chunk_relation.chunk if context.prefetch == "auto": @@ -185,7 +188,7 @@ class ItemBatchIn(BaseModel): @sync_to_async def collection_list_common( queryset: QuerySet, - user: User, + user: UserType, stoken: t.Optional[str], limit: int, prefetch: Prefetch, @@ -210,7 +213,7 @@ def collection_list_common( remed = remed_qs.values_list("collection__uid", flat=True) if len(remed) > 0: - ret.removedMemberships = [{"uid": x} for x in remed] + ret.removedMemberships = [RemovedMembershipOut(uid=x) for x in remed] return ret @@ -219,14 +222,14 @@ def collection_list_common( def verify_collection_admin( - collection: models.Collection = Depends(get_collection), user: User = Depends(get_authenticated_user) + collection: models.Collection = Depends(get_collection), user: UserType = Depends(get_authenticated_user) ): if not is_collection_admin(collection, user): raise PermissionDenied("admin_access_required", "Only collection admins can perform this operation.") def has_write_access( - collection: models.Collection = Depends(get_collection), user: User = Depends(get_authenticated_user) + collection: models.Collection = Depends(get_collection), user: UserType = Depends(get_authenticated_user) ): member = collection.members.get(user=user) if member.accessLevel == models.AccessLevels.READ_ONLY: @@ -247,7 +250,7 @@ async def list_multi( stoken: t.Optional[str] = None, limit: int = 50, queryset: QuerySet = Depends(get_collection_queryset), - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") @@ -263,7 +266,7 @@ async def collection_list( stoken: t.Optional[str] = None, limit: int = 50, prefetch: Prefetch = PrefetchQuery, - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), queryset: QuerySet = Depends(get_collection_queryset), ): return await collection_list_common(queryset, user, stoken, limit, prefetch) @@ -299,7 +302,7 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle return revision -def _create(data: CollectionIn, user: User): +def _create(data: CollectionIn, user: UserType): with transaction.atomic(): if data.item.etag is not None: raise ValidationError("bad_etag", "etag is not null") @@ -335,14 +338,14 @@ def _create(data: CollectionIn, user: User): @collection_router.post("/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE) -async def create(data: CollectionIn, user: User = Depends(get_authenticated_user)): +async def create(data: CollectionIn, user: UserType = Depends(get_authenticated_user)): await sync_to_async(_create)(data, user) @collection_router.get("/{collection_uid}/", response_model=CollectionOut, dependencies=PERMISSIONS_READ) def collection_get( obj: models.Collection = Depends(get_collection), - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): return CollectionOut.from_orm_context(obj, Context(user, prefetch)) @@ -393,7 +396,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val def item_get( item_uid: str, queryset: QuerySet = Depends(get_item_queryset), - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): obj = queryset.get(uid=item_uid) @@ -403,7 +406,7 @@ def item_get( @sync_to_async def item_list_common( queryset: QuerySet, - user: User, + user: UserType, stoken: t.Optional[str], limit: int, prefetch: Prefetch, @@ -424,7 +427,7 @@ async def item_list( limit: int = 50, prefetch: Prefetch = PrefetchQuery, withCollection: bool = False, - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), ): if not withCollection: queryset = queryset.filter(parent__isnull=True) @@ -433,7 +436,7 @@ async def item_list( return response -def item_bulk_common(data: ItemBatchIn, user: User, stoken: t.Optional[str], uid: str, validate_etag: bool): +def item_bulk_common(data: ItemBatchIn, user: UserType, stoken: t.Optional[str], uid: str, validate_etag: bool): queryset = get_collection_queryset(user) with transaction.atomic(): # We need this for locking the collection object collection_object = queryset.select_for_update().get(uid=uid) @@ -467,7 +470,7 @@ def item_revisions( limit: int = 50, iterator: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), items: QuerySet = Depends(get_item_queryset), ): item = get_object_or_404(items, uid=item_uid) @@ -501,7 +504,7 @@ def fetch_updates( data: t.List[CollectionItemBulkGetIn], stoken: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), queryset: QuerySet = Depends(get_item_queryset), ): # FIXME: make configurable? @@ -531,14 +534,14 @@ def fetch_updates( @item_router.post("/item/transaction/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_transaction( - collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) + collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: UserType = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) @item_router.post("/item/batch/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_batch( - collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: User = Depends(get_authenticated_user) + collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: UserType = Depends(get_authenticated_user) ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) diff --git a/etebase_fastapi/dependencies.py b/etebase_fastapi/dependencies.py index ddb9b3b..fb9cec5 100644 --- a/etebase_fastapi/dependencies.py +++ b/etebase_fastapi/dependencies.py @@ -3,17 +3,17 @@ import dataclasses from fastapi import Depends from fastapi.security import APIKeyHeader -from django.contrib.auth import get_user_model from django.utils import timezone from django.db.models import QuerySet from django_etebase import models from django_etebase.token_auth.models import AuthToken, get_default_expiry +from myauth.models import UserType, get_typed_user_model from .exceptions import AuthenticationFailed from .utils import get_object_or_404 -User = get_user_model() +User = get_typed_user_model() token_scheme = APIKeyHeader(name="Authorization") AUTO_REFRESH = True MIN_REFRESH_INTERVAL = 60 @@ -21,7 +21,7 @@ MIN_REFRESH_INTERVAL = 60 @dataclasses.dataclass(frozen=True) class AuthData: - user: User + user: UserType token: AuthToken @@ -60,12 +60,12 @@ def get_auth_data(api_token: str = Depends(token_scheme)) -> AuthData: return AuthData(user, token) -def get_authenticated_user(api_token: str = Depends(token_scheme)) -> User: +def get_authenticated_user(api_token: str = Depends(token_scheme)) -> UserType: user, _ = __get_authenticated_user(api_token) return user -def get_collection_queryset(user: User = Depends(get_authenticated_user)) -> QuerySet: +def get_collection_queryset(user: UserType = Depends(get_authenticated_user)) -> QuerySet: default_queryset: QuerySet = models.Collection.objects.all() return default_queryset.filter(members__user=user) diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/invitation.py index 9e731bc..eb9f549 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/invitation.py @@ -1,12 +1,12 @@ import typing as t -from django.contrib.auth import get_user_model from django.db import transaction, IntegrityError from django.db.models import QuerySet from fastapi import APIRouter, Depends, status, Request from django_etebase import models from django_etebase.utils import get_user_queryset, CallbackContext +from myauth.models import UserType, get_typed_user_model from .authentication import get_authenticated_user from .exceptions import HttpError, PermissionDenied from .msgpack import MsgpackRoute @@ -20,7 +20,7 @@ from .utils import ( PERMISSIONS_READWRITE, ) -User = get_user_model() +User = get_typed_user_model() invitation_incoming_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) invitation_outgoing_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) default_queryset: QuerySet = models.CollectionInvitation.objects.all() @@ -53,7 +53,8 @@ class CollectionInvitationCommon(BaseModel): class CollectionInvitationIn(CollectionInvitationCommon): def validate_db(self, context: Context): - if context.user.username == self.username.lower(): + user = context.user + if user is not None and (user.username == self.username.lower()): raise HttpError("no_self_invite", "Inviting yourself is not allowed") @@ -84,11 +85,11 @@ class InvitationListResponse(BaseModel): done: bool -def get_incoming_queryset(user: User = Depends(get_authenticated_user)): +def get_incoming_queryset(user: UserType = Depends(get_authenticated_user)): return default_queryset.filter(user=user) -def get_outgoing_queryset(user: User = Depends(get_authenticated_user)): +def get_outgoing_queryset(user: UserType = Depends(get_authenticated_user)): return default_queryset.filter(fromMember__user=user) @@ -183,7 +184,7 @@ def incoming_accept( def outgoing_create( data: CollectionInvitationIn, request: Request, - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), ): collection = get_object_or_404(models.Collection.objects, uid=data.collection) to_user = get_object_or_404( @@ -231,7 +232,7 @@ def outgoing_delete( def outgoing_fetch_user_profile( username: str, request: Request, - user: User = Depends(get_authenticated_user), + user: UserType = Depends(get_authenticated_user), ): kwargs = {User.USERNAME_FIELD: username.lower()} user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) diff --git a/etebase_fastapi/member.py b/etebase_fastapi/member.py index 725d44b..22977ac 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/member.py @@ -1,11 +1,11 @@ import typing as t -from django.contrib.auth import get_user_model from django.db import transaction from django.db.models import QuerySet from fastapi import APIRouter, Depends, status from django_etebase import models +from myauth.models import UserType, get_typed_user_model from .authentication import get_authenticated_user from .msgpack import MsgpackRoute from .utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE @@ -13,7 +13,7 @@ from .stoken_handler import filter_by_stoken_and_limit from .collection import get_collection, verify_collection_admin -User = get_user_model() +User = get_typed_user_model() member_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) default_queryset: QuerySet = models.CollectionMember.objects.all() @@ -98,6 +98,8 @@ def member_patch( @member_router.post("/member/leave/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READ) -def member_leave(user: User = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection)): +def member_leave( + user: UserType = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection) +): obj = get_object_or_404(collection.members, user=user) obj.revoke() diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index edffd7e..915e783 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -19,13 +19,15 @@ class MsgpackRequest(Request): class MsgpackResponse(Response): media_type = "application/msgpack" - def render(self, content: t.Optional[t.Any]) -> t.Optional[bytes]: + def render(self, content: t.Optional[t.Any]) -> bytes: if content is None: return b"" if isinstance(content, BaseModel): content = content.dict() - return msgpack.packb(content, use_bin_type=True) + ret = msgpack.packb(content, use_bin_type=True) + assert ret is not None + return ret class MsgpackRoute(APIRoute): diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/test_reset_view.py index 3075290..e328875 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/test_reset_view.py @@ -1,5 +1,4 @@ from django.conf import settings -from django.contrib.auth import get_user_model from django.db import transaction from django.shortcuts import get_object_or_404 from fastapi import APIRouter, Request, status @@ -8,9 +7,10 @@ from django_etebase.utils import get_user_queryset, CallbackContext from etebase_fastapi.authentication import SignupIn, signup_save from etebase_fastapi.msgpack import MsgpackRoute from etebase_fastapi.exceptions import HttpError +from myauth.models import get_typed_user_model test_reset_view_router = APIRouter(route_class=MsgpackRoute, tags=["test helpers"]) -User = get_user_model() +User = get_typed_user_model() @test_reset_view_router.post("/reset/", status_code=status.HTTP_204_NO_CONTENT) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 7280018..c91c3ec 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -8,14 +8,14 @@ from pydantic import BaseModel as PyBaseModel from django.db.models import QuerySet from django.core.exceptions import ObjectDoesNotExist -from django.contrib.auth import get_user_model from django_etebase import app_settings from django_etebase.models import AccessLevels +from myauth.models import UserType, get_typed_user_model from .exceptions import HttpError, HttpErrorOut -User = get_user_model() +User = get_typed_user_model() Prefetch = t.Literal["auto", "medium"] PrefetchQuery = Query(default="auto") @@ -30,7 +30,7 @@ class BaseModel(PyBaseModel): @dataclasses.dataclass class Context: - user: t.Optional[User] + user: t.Optional[UserType] prefetch: t.Optional[Prefetch] diff --git a/myauth/forms.py b/myauth/forms.py index 7aacb9b..fc2be74 100644 --- a/myauth/forms.py +++ b/myauth/forms.py @@ -1,8 +1,8 @@ from django import forms -from django.contrib.auth import get_user_model from django.contrib.auth.forms import UsernameField +from myauth.models import get_typed_user_model -User = get_user_model() +User = get_typed_user_model() class AdminUserCreationForm(forms.ModelForm): diff --git a/myauth/models.py b/myauth/models.py index d6585a8..5bc4af7 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -1,3 +1,5 @@ +import typing as t + from django.contrib.auth.models import AbstractUser, UserManager as DjangoUserManager from django.core import validators from django.db import models @@ -28,9 +30,21 @@ class User(AbstractUser): unique=True, help_text=_("Required. 150 characters or fewer. Letters, digits and ./-/_ only."), validators=[username_validator], - error_messages={"unique": _("A user with that username already exists."),}, + error_messages={ + "unique": _("A user with that username already exists."), + }, ) @classmethod def normalize_username(cls, username): return super().normalize_username(username).lower() + + +UserType = t.Type[User] + + +def get_typed_user_model() -> UserType: + from django.contrib.auth import get_user_model + + ret: t.Any = get_user_model() + return ret From 84b6114e99c206e56292386035dfc7ef0ac84ed4 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 13:43:11 +0200 Subject: [PATCH 455/511] Requirements: add dev requirements and django-stubs. --- requirements-dev.txt | 28 ++++++++++++++++++++++++++++ requirements.in/development.txt | 3 ++- 2 files changed, 30 insertions(+), 1 deletion(-) create mode 100644 requirements-dev.txt diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..15a8b60 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,28 @@ +# +# This file is autogenerated by pip-compile +# To update, run: +# +# pip-compile --output-file=requirements-dev.txt requirements.in/development.txt +# +appdirs==1.4.4 # via black +asgiref==3.3.1 # via django +black==20.8b1 # via -r requirements.in/development.txt +click==7.1.2 # via black, pip-tools +coverage==5.3.1 # via -r requirements.in/development.txt +django-stubs==1.7.0 # via -r requirements.in/development.txt +django==3.1.4 # via django-stubs +mypy-extensions==0.4.3 # via black, mypy +mypy==0.790 # via django-stubs +pathspec==0.8.1 # via black +pip-tools==5.4.0 # via -r requirements.in/development.txt +pytz==2020.5 # via django +pywatchman==1.4.1 # via -r requirements.in/development.txt +regex==2020.11.13 # via black +six==1.15.0 # via pip-tools +sqlparse==0.4.1 # via django +toml==0.10.2 # via black +typed-ast==1.4.1 # via black, mypy +typing-extensions==3.7.4.3 # via black, django-stubs, mypy + +# The following packages are considered to be unsafe in a requirements file: +# pip diff --git a/requirements.in/development.txt b/requirements.in/development.txt index a956471..fb281d3 100644 --- a/requirements.in/development.txt +++ b/requirements.in/development.txt @@ -1,4 +1,5 @@ coverage pip-tools pywatchman -black \ No newline at end of file +black +django-stubs From ff55904f49f32bd21c17f5d383149891e24458eb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 14:04:17 +0200 Subject: [PATCH 456/511] Fix user type --- myauth/models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/myauth/models.py b/myauth/models.py index 5bc4af7..7786bf0 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -40,7 +40,7 @@ class User(AbstractUser): return super().normalize_username(username).lower() -UserType = t.Type[User] +UserType = User def get_typed_user_model() -> UserType: From e6b47ae1a97fb025c6790a201ec201bc14113f4d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 15:17:46 +0200 Subject: [PATCH 457/511] Fix login_challenge to work with get_user_queryset. --- etebase_fastapi/authentication.py | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/authentication.py index 064d2da..e77cdb8 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/authentication.py @@ -111,11 +111,13 @@ class SignupIn(BaseModel): encryptedContent: bytes -@sync_to_async -def __get_login_user(username: str) -> UserType: +def get_login_user(request: Request, challenge: LoginChallengeIn) -> UserType: + username = challenge.username + kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} try: - user = User.objects.get(**kwargs) + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) + user = user_queryset.get(**kwargs) if not hasattr(user, "userinfo"): raise AuthenticationFailed(code="user_not_init", detail="User not properly init") return user @@ -123,11 +125,6 @@ def __get_login_user(username: str) -> UserType: raise AuthenticationFailed(code="user_not_found", detail="User not found") -async def get_login_user(challenge: LoginChallengeIn) -> UserType: - user = await __get_login_user(challenge.username) - return user - - def get_encryption_key(salt): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) return nacl.hash.blake2b( @@ -196,7 +193,7 @@ def login_challenge(user: UserType = Depends(get_login_user)): @authentication_router.post("/login/", response_model=LoginOut) async def login(data: Login, request: Request): - user = await get_login_user(LoginChallengeIn(username=data.response_data.username)) + user = await sync_to_async(get_login_user)(request, LoginChallengeIn(username=data.response_data.username)) host = request.headers.get("Host") await validate_login_request(data.response_data, data, user, "login", host) data = await sync_to_async(LoginOut.from_orm)(user) From 8bf04fc28638609f3b11559409738ffba6c851e2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 15:37:11 +0200 Subject: [PATCH 458/511] Reformat files using black. --- etebase_fastapi/collection.py | 10 ++++++++-- etebase_fastapi/sendfile/utils.py | 4 +++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/collection.py index 9e25b38..14abee9 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/collection.py @@ -534,14 +534,20 @@ def fetch_updates( @item_router.post("/item/transaction/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_transaction( - collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: UserType = Depends(get_authenticated_user) + collection_uid: str, + data: ItemBatchIn, + stoken: t.Optional[str] = None, + user: UserType = Depends(get_authenticated_user), ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) @item_router.post("/item/batch/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_batch( - collection_uid: str, data: ItemBatchIn, stoken: t.Optional[str] = None, user: UserType = Depends(get_authenticated_user) + collection_uid: str, + data: ItemBatchIn, + stoken: t.Optional[str] = None, + user: UserType = Depends(get_authenticated_user), ): return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) diff --git a/etebase_fastapi/sendfile/utils.py b/etebase_fastapi/sendfile/utils.py index 7c8b1f2..c35d6df 100644 --- a/etebase_fastapi/sendfile/utils.py +++ b/etebase_fastapi/sendfile/utils.py @@ -56,7 +56,9 @@ def _sanitize_path(filepath): try: filepath_abs.relative_to(path_root) except ValueError: - raise HttpError("generic", "{} wrt {} is impossible".format(filepath_abs, path_root), status_code=status.HTTP_404_NOT_FOUND) + raise HttpError( + "generic", "{} wrt {} is impossible".format(filepath_abs, path_root), status_code=status.HTTP_404_NOT_FOUND + ) return filepath_abs From dbdff06e686ccba1649f77539e4cf0a48bf9b8f5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 15:42:41 +0200 Subject: [PATCH 459/511] Move all the routers under their own dir. --- etebase_fastapi/main.py | 10 +++++----- etebase_fastapi/routers/__init__.py | 0 etebase_fastapi/{ => routers}/authentication.py | 8 ++++---- etebase_fastapi/{ => routers}/collection.py | 12 ++++++------ etebase_fastapi/{ => routers}/invitation.py | 6 +++--- etebase_fastapi/{ => routers}/member.py | 6 +++--- etebase_fastapi/{ => routers}/test_reset_view.py | 6 +++--- 7 files changed, 24 insertions(+), 24 deletions(-) create mode 100644 etebase_fastapi/routers/__init__.py rename etebase_fastapi/{ => routers}/authentication.py (97%) rename etebase_fastapi/{ => routers}/collection.py (98%) rename etebase_fastapi/{ => routers}/invitation.py (98%) rename etebase_fastapi/{ => routers}/member.py (94%) rename etebase_fastapi/{ => routers}/test_reset_view.py (90%) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index 534798e..8e8469c 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -6,11 +6,11 @@ from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware from .exceptions import CustomHttpException -from .authentication import authentication_router -from .collection import collection_router, item_router -from .member import member_router -from .invitation import invitation_incoming_router, invitation_outgoing_router from .msgpack import MsgpackResponse +from .routers.authentication import authentication_router +from .routers.collection import collection_router, item_router +from .routers.member import member_router +from .routers.invitation import invitation_incoming_router, invitation_outgoing_router def create_application(prefix="", middlewares=[]): @@ -38,7 +38,7 @@ def create_application(prefix="", middlewares=[]): ) if settings.DEBUG: - from etebase_fastapi.test_reset_view import test_reset_view_router + from etebase_fastapi.routers.test_reset_view import test_reset_view_router app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") diff --git a/etebase_fastapi/routers/__init__.py b/etebase_fastapi/routers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/etebase_fastapi/authentication.py b/etebase_fastapi/routers/authentication.py similarity index 97% rename from etebase_fastapi/authentication.py rename to etebase_fastapi/routers/authentication.py index e77cdb8..4b1b57a 100644 --- a/etebase_fastapi/authentication.py +++ b/etebase_fastapi/routers/authentication.py @@ -20,10 +20,10 @@ from django_etebase.models import UserInfo from django_etebase.signals import user_signed_up from django_etebase.utils import create_user, get_user_queryset, CallbackContext from myauth.models import UserType, get_typed_user_model -from .exceptions import AuthenticationFailed, transform_validation_error, HttpError -from .msgpack import MsgpackRoute -from .utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode -from .dependencies import AuthData, get_auth_data, get_authenticated_user +from ..exceptions import AuthenticationFailed, transform_validation_error, HttpError +from ..msgpack import MsgpackRoute +from ..utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode +from ..dependencies import AuthData, get_auth_data, get_authenticated_user User = get_typed_user_model() authentication_router = APIRouter(route_class=MsgpackRoute) diff --git a/etebase_fastapi/collection.py b/etebase_fastapi/routers/collection.py similarity index 98% rename from etebase_fastapi/collection.py rename to etebase_fastapi/routers/collection.py index 14abee9..56afd7b 100644 --- a/etebase_fastapi/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -10,10 +10,10 @@ from fastapi import APIRouter, Depends, status, Request from django_etebase import models from myauth.models import UserType, get_typed_user_model from .authentication import get_authenticated_user -from .exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError -from .msgpack import MsgpackRoute -from .stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken -from .utils import ( +from ..exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError +from ..msgpack import MsgpackRoute +from ..stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken +from ..utils import ( get_object_or_404, Context, Prefetch, @@ -24,8 +24,8 @@ from .utils import ( PERMISSIONS_READ, PERMISSIONS_READWRITE, ) -from .dependencies import get_collection_queryset, get_item_queryset, get_collection -from .sendfile import sendfile +from ..dependencies import get_collection_queryset, get_item_queryset, get_collection +from ..sendfile import sendfile User = get_typed_user_model collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) diff --git a/etebase_fastapi/invitation.py b/etebase_fastapi/routers/invitation.py similarity index 98% rename from etebase_fastapi/invitation.py rename to etebase_fastapi/routers/invitation.py index eb9f549..6a06c60 100644 --- a/etebase_fastapi/invitation.py +++ b/etebase_fastapi/routers/invitation.py @@ -8,9 +8,9 @@ from django_etebase import models from django_etebase.utils import get_user_queryset, CallbackContext from myauth.models import UserType, get_typed_user_model from .authentication import get_authenticated_user -from .exceptions import HttpError, PermissionDenied -from .msgpack import MsgpackRoute -from .utils import ( +from ..exceptions import HttpError, PermissionDenied +from ..msgpack import MsgpackRoute +from ..utils import ( get_object_or_404, Context, is_collection_admin, diff --git a/etebase_fastapi/member.py b/etebase_fastapi/routers/member.py similarity index 94% rename from etebase_fastapi/member.py rename to etebase_fastapi/routers/member.py index 22977ac..210374c 100644 --- a/etebase_fastapi/member.py +++ b/etebase_fastapi/routers/member.py @@ -7,9 +7,9 @@ from fastapi import APIRouter, Depends, status from django_etebase import models from myauth.models import UserType, get_typed_user_model from .authentication import get_authenticated_user -from .msgpack import MsgpackRoute -from .utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE -from .stoken_handler import filter_by_stoken_and_limit +from ..msgpack import MsgpackRoute +from ..utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE +from ..stoken_handler import filter_by_stoken_and_limit from .collection import get_collection, verify_collection_admin diff --git a/etebase_fastapi/test_reset_view.py b/etebase_fastapi/routers/test_reset_view.py similarity index 90% rename from etebase_fastapi/test_reset_view.py rename to etebase_fastapi/routers/test_reset_view.py index e328875..09638e4 100644 --- a/etebase_fastapi/test_reset_view.py +++ b/etebase_fastapi/routers/test_reset_view.py @@ -4,9 +4,9 @@ from django.shortcuts import get_object_or_404 from fastapi import APIRouter, Request, status from django_etebase.utils import get_user_queryset, CallbackContext -from etebase_fastapi.authentication import SignupIn, signup_save -from etebase_fastapi.msgpack import MsgpackRoute -from etebase_fastapi.exceptions import HttpError +from .authentication import SignupIn, signup_save +from ..msgpack import MsgpackRoute +from ..exceptions import HttpError from myauth.models import get_typed_user_model test_reset_view_router = APIRouter(route_class=MsgpackRoute, tags=["test helpers"]) From 62eb46ec4ef036673b1864c871aa477cf6fde002 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 15:44:52 +0200 Subject: [PATCH 460/511] msgpack route: use the encode/decode functions from the utils module. --- etebase_fastapi/msgpack.py | 9 ++++----- etebase_fastapi/utils.py | 4 +++- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index 915e783..8de8806 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -1,10 +1,11 @@ import typing as t -import msgpack from fastapi.routing import APIRoute, get_request_handler from pydantic import BaseModel from starlette.requests import Request from starlette.responses import Response +from .utils import msgpack_encode, msgpack_decode + class MsgpackRequest(Request): media_type = "application/msgpack" @@ -12,7 +13,7 @@ class MsgpackRequest(Request): async def json(self) -> bytes: if not hasattr(self, "_json"): body = await super().body() - self._json = msgpack.unpackb(body, raw=False) + self._json = msgpack_decode(body) return self._json @@ -25,9 +26,7 @@ class MsgpackResponse(Response): if isinstance(content, BaseModel): content = content.dict() - ret = msgpack.packb(content, use_bin_type=True) - assert ret is not None - return ret + return msgpack_encode(content) class MsgpackRoute(APIRoute): diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index c91c3ec..3a091c5 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -47,7 +47,9 @@ def is_collection_admin(collection, user): def msgpack_encode(content): - return msgpack.packb(content, use_bin_type=True) + ret = msgpack.packb(content, use_bin_type=True) + assert ret is not None + return ret def msgpack_decode(content): From 174e54681d8688a2c048ce3d199dea07ae99354d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 16:06:59 +0200 Subject: [PATCH 461/511] Improve type annotations. --- django_etebase/models.py | 1 + myauth/models.py | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index 096371d..3060fa4 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -124,6 +124,7 @@ def generate_stoken_uid(): class Stoken(models.Model): + id: int uid = models.CharField( db_index=True, unique=True, diff --git a/myauth/models.py b/myauth/models.py index 7786bf0..c9298a4 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -20,9 +20,10 @@ class UserManager(DjangoUserManager): class User(AbstractUser): + id: int username_validator = UnicodeUsernameValidator() - objects = UserManager() + objects: UserManager = UserManager() username = models.CharField( _("username"), From 8245577dfb1117eab0cac81c4c4224b39967d757 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 16:55:21 +0200 Subject: [PATCH 462/511] Rename module to prevent confusion on import. --- django_etebase/__init__.py | 2 +- django_etebase/{app_settings.py => app_settings_inner.py} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename django_etebase/{app_settings.py => app_settings_inner.py} (100%) diff --git a/django_etebase/__init__.py b/django_etebase/__init__.py index 426fefd..99ee8b6 100644 --- a/django_etebase/__init__.py +++ b/django_etebase/__init__.py @@ -1 +1 @@ -from .app_settings import app_settings +from .app_settings_inner import app_settings diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings_inner.py similarity index 100% rename from django_etebase/app_settings.py rename to django_etebase/app_settings_inner.py From 332f7e2332757e7aa3aa8fec50583679be1a8608 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 17:52:08 +0200 Subject: [PATCH 463/511] Fix Python 3.7 compatibility Both cached_property and Literal were introduced in Python 3.8 so they can't be used. --- etebase_fastapi/routers/authentication.py | 5 +++-- etebase_fastapi/utils.py | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/etebase_fastapi/routers/authentication.py b/etebase_fastapi/routers/authentication.py index 4b1b57a..efab7ef 100644 --- a/etebase_fastapi/routers/authentication.py +++ b/etebase_fastapi/routers/authentication.py @@ -1,6 +1,6 @@ import typing as t +from typing_extensions import Literal from datetime import datetime -from functools import cached_property import nacl import nacl.encoding @@ -12,6 +12,7 @@ from django.conf import settings from django.contrib.auth import user_logged_out, user_logged_in from django.core import exceptions as django_exceptions from django.db import transaction +from django.utils.functional import cached_property from fastapi import APIRouter, Depends, status, Request from django_etebase import app_settings, models @@ -43,7 +44,7 @@ class LoginResponse(BaseModel): username: str challenge: bytes host: str - action: t.Literal["login", "changePassword"] + action: Literal["login", "changePassword"] class UserOut(BaseModel): diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 3a091c5..03f1a7d 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -1,5 +1,6 @@ import dataclasses import typing as t +from typing_extensions import Literal import msgpack import base64 @@ -17,7 +18,7 @@ from .exceptions import HttpError, HttpErrorOut User = get_typed_user_model() -Prefetch = t.Literal["auto", "medium"] +Prefetch = Literal["auto", "medium"] PrefetchQuery = Query(default="auto") From 709bc6c1fc2be28b6d2ad9e4bc7bd12de2828dfd Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 17:18:09 +0200 Subject: [PATCH 464/511] Improve typing information. --- django_etebase/models.py | 47 ++++++++++++++++++++------- django_etebase/utils.py | 5 +-- etebase_fastapi/routers/collection.py | 18 +++++----- etebase_fastapi/routers/invitation.py | 19 ++++++----- etebase_fastapi/routers/member.py | 9 ++--- etebase_fastapi/utils.py | 7 ++-- myauth/models.py | 4 +-- 7 files changed, 70 insertions(+), 39 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index 3060fa4..7725a19 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -12,6 +12,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . +import typing as t from pathlib import Path from django.db import models, transaction @@ -28,7 +29,7 @@ from . import app_settings UidValidator = RegexValidator(regex=r"^[a-zA-Z0-9\-_]{20,}$", message="Not a valid UID") -def stoken_annotation_builder(stoken_id_fields): +def stoken_annotation_builder(stoken_id_fields: t.List[str]): aggr_fields = [Coalesce(Max(field), V(0)) for field in stoken_id_fields] return Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] @@ -37,6 +38,8 @@ class CollectionType(models.Model): owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True) + objects: models.manager.BaseManager["CollectionType"] + class Collection(models.Model): main_item = models.OneToOneField("CollectionItem", related_name="parent", null=True, on_delete=models.SET_NULL) @@ -46,19 +49,21 @@ class Collection(models.Model): stoken_annotation = stoken_annotation_builder(["items__revisions__stoken", "members__stoken"]) + objects: models.manager.BaseManager["Collection"] + def __str__(self): return self.uid @property - def content(self): + def content(self) -> "CollectionItemRevision": return self.main_item.content @property - def etag(self): + def etag(self) -> str: return self.content.uid @cached_property - def stoken(self): + def stoken(self) -> str: stoken_id = ( self.__class__.objects.filter(main_item=self.main_item) .annotate(max_stoken=self.stoken_annotation) @@ -80,6 +85,8 @@ class CollectionItem(models.Model): stoken_annotation = stoken_annotation_builder(["revisions__stoken"]) + objects: models.manager.BaseManager["CollectionItem"] + class Meta: unique_together = ("uid", "collection") @@ -87,23 +94,23 @@ class CollectionItem(models.Model): return "{} {}".format(self.uid, self.collection.uid) @cached_property - def content(self): + def content(self) -> "CollectionItemRevision": return self.revisions.get(current=True) @property - def etag(self): + def etag(self) -> str: return self.content.uid -def chunk_directory_path(instance, filename): +def chunk_directory_path(instance: "CollectionItemChunk", filename: str) -> Path: custom_func = app_settings.CHUNK_PATH_FUNC if custom_func is not None: return custom_func(instance, filename) - col = instance.collection - user_id = col.owner.id - uid_prefix = instance.uid[:2] - uid_rest = instance.uid[2:] + col: Collection = instance.collection + user_id: int = col.owner.id + uid_prefix: str = instance.uid[:2] + uid_rest: str = instance.uid[2:] return Path("user_{}".format(user_id), col.uid, uid_prefix, uid_rest) @@ -112,6 +119,8 @@ class CollectionItemChunk(models.Model): collection = models.ForeignKey(Collection, related_name="chunks", on_delete=models.CASCADE) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) + objects: models.manager.BaseManager["CollectionItemChunk"] + def __str__(self): return self.uid @@ -135,6 +144,8 @@ class Stoken(models.Model): validators=[UidValidator], ) + objects: models.manager.BaseManager["Stoken"] + class CollectionItemRevision(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) @@ -146,6 +157,8 @@ class CollectionItemRevision(models.Model): current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) + objects: models.manager.BaseManager["CollectionItemRevision"] + class Meta: unique_together = ("item", "current") @@ -157,6 +170,8 @@ class RevisionChunkRelation(models.Model): chunk = models.ForeignKey(CollectionItemChunk, related_name="revisions_relation", on_delete=models.CASCADE) revision = models.ForeignKey(CollectionItemRevision, related_name="chunks_relation", on_delete=models.CASCADE) + objects: models.manager.BaseManager["RevisionChunkRelation"] + class Meta: ordering = ("id",) @@ -180,6 +195,8 @@ class CollectionMember(models.Model): stoken_annotation = stoken_annotation_builder(["stoken"]) + objects: models.manager.BaseManager["CollectionMember"] + class Meta: unique_together = ("user", "collection") @@ -204,6 +221,8 @@ class CollectionMemberRemoved(models.Model): collection = models.ForeignKey(Collection, related_name="removed_members", on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + objects: models.manager.BaseManager["CollectionMemberRemoved"] + class Meta: unique_together = ("user", "collection") @@ -225,6 +244,8 @@ class CollectionInvitation(models.Model): default=AccessLevels.READ_ONLY, ) + objects: models.manager.BaseManager["CollectionInvitation"] + class Meta: unique_together = ("user", "fromMember") @@ -232,7 +253,7 @@ class CollectionInvitation(models.Model): return "{} {}".format(self.fromMember.collection.uid, self.user) @cached_property - def collection(self): + def collection(self) -> Collection: return self.fromMember.collection @@ -244,5 +265,7 @@ class UserInfo(models.Model): encryptedContent = models.BinaryField(editable=True, blank=False, null=False) salt = models.BinaryField(editable=True, blank=False, null=False) + objects: models.manager.BaseManager["UserInfo"] + def __str__(self): return "UserInfo<{}>".format(self.owner) diff --git a/django_etebase/utils.py b/django_etebase/utils.py index d812ae3..3a05fd4 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -1,6 +1,7 @@ import typing as t from dataclasses import dataclass +from django.db.models import QuerySet from django.core.exceptions import PermissionDenied from myauth.models import UserType, get_typed_user_model @@ -18,14 +19,14 @@ class CallbackContext: user: t.Optional[UserType] = None -def get_user_queryset(queryset, context: CallbackContext): +def get_user_queryset(queryset: QuerySet[UserType], context: CallbackContext) -> QuerySet[UserType]: custom_func = app_settings.GET_USER_QUERYSET_FUNC if custom_func is not None: return custom_func(queryset, context) return queryset -def create_user(context: CallbackContext, *args, **kwargs): +def create_user(context: CallbackContext, *args, **kwargs) -> UserType: custom_func = app_settings.CREATE_USER_FUNC if custom_func is not None: return custom_func(context, *args, **kwargs) diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index 56afd7b..4825626 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -30,6 +30,8 @@ from ..sendfile import sendfile User = get_typed_user_model collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +CollectionQuerySet = QuerySet[models.Collection] +CollectionItemQuerySet = QuerySet[models.CollectionItem] class ListMulti(BaseModel): @@ -187,7 +189,7 @@ class ItemBatchIn(BaseModel): @sync_to_async def collection_list_common( - queryset: QuerySet, + queryset: CollectionQuerySet, user: UserType, stoken: t.Optional[str], limit: int, @@ -249,7 +251,7 @@ async def list_multi( data: ListMulti, stoken: t.Optional[str] = None, limit: int = 50, - queryset: QuerySet = Depends(get_collection_queryset), + queryset: CollectionQuerySet = Depends(get_collection_queryset), user: UserType = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): @@ -267,7 +269,7 @@ async def collection_list( limit: int = 50, prefetch: Prefetch = PrefetchQuery, user: UserType = Depends(get_authenticated_user), - queryset: QuerySet = Depends(get_collection_queryset), + queryset: CollectionQuerySet = Depends(get_collection_queryset), ): return await collection_list_common(queryset, user, stoken, limit, prefetch) @@ -395,7 +397,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val @item_router.get("/item/{item_uid}/", response_model=CollectionItemOut, dependencies=PERMISSIONS_READ) def item_get( item_uid: str, - queryset: QuerySet = Depends(get_item_queryset), + queryset: CollectionItemQuerySet = Depends(get_item_queryset), user: UserType = Depends(get_authenticated_user), prefetch: Prefetch = PrefetchQuery, ): @@ -405,7 +407,7 @@ def item_get( @sync_to_async def item_list_common( - queryset: QuerySet, + queryset: CollectionItemQuerySet, user: UserType, stoken: t.Optional[str], limit: int, @@ -422,7 +424,7 @@ def item_list_common( @item_router.get("/item/", response_model=CollectionItemListResponse, dependencies=PERMISSIONS_READ) async def item_list( - queryset: QuerySet = Depends(get_item_queryset), + queryset: CollectionItemQuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, limit: int = 50, prefetch: Prefetch = PrefetchQuery, @@ -471,7 +473,7 @@ def item_revisions( iterator: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, user: UserType = Depends(get_authenticated_user), - items: QuerySet = Depends(get_item_queryset), + items: CollectionItemQuerySet = Depends(get_item_queryset), ): item = get_object_or_404(items, uid=item_uid) @@ -505,7 +507,7 @@ def fetch_updates( stoken: t.Optional[str] = None, prefetch: Prefetch = PrefetchQuery, user: UserType = Depends(get_authenticated_user), - queryset: QuerySet = Depends(get_item_queryset), + queryset: CollectionItemQuerySet = Depends(get_item_queryset), ): # FIXME: make configurable? item_limit = 200 diff --git a/etebase_fastapi/routers/invitation.py b/etebase_fastapi/routers/invitation.py index 6a06c60..aceb05d 100644 --- a/etebase_fastapi/routers/invitation.py +++ b/etebase_fastapi/routers/invitation.py @@ -23,7 +23,8 @@ from ..utils import ( User = get_typed_user_model() invitation_incoming_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) invitation_outgoing_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) -default_queryset: QuerySet = models.CollectionInvitation.objects.all() +InvitationQuerySet = QuerySet[models.CollectionInvitation] +default_queryset: InvitationQuerySet = models.CollectionInvitation.objects.all() class UserInfoOut(BaseModel): @@ -94,7 +95,7 @@ def get_outgoing_queryset(user: UserType = Depends(get_authenticated_user)): def list_common( - queryset: QuerySet, + queryset: InvitationQuerySet, iterator: t.Optional[str], limit: int, ) -> InvitationListResponse: @@ -125,7 +126,7 @@ def list_common( def incoming_list( iterator: t.Optional[str] = None, limit: int = 50, - queryset: QuerySet = Depends(get_incoming_queryset), + queryset: InvitationQuerySet = Depends(get_incoming_queryset), ): return list_common(queryset, iterator, limit) @@ -135,7 +136,7 @@ def incoming_list( ) def incoming_get( invitation_uid: str, - queryset: QuerySet = Depends(get_incoming_queryset), + queryset: InvitationQuerySet = Depends(get_incoming_queryset), ): obj = get_object_or_404(queryset, uid=invitation_uid) return CollectionInvitationOut.from_orm(obj) @@ -146,7 +147,7 @@ def incoming_get( ) def incoming_delete( invitation_uid: str, - queryset: QuerySet = Depends(get_incoming_queryset), + queryset: InvitationQuerySet = Depends(get_incoming_queryset), ): obj = get_object_or_404(queryset, uid=invitation_uid) obj.delete() @@ -158,7 +159,7 @@ def incoming_delete( def incoming_accept( invitation_uid: str, data: CollectionInvitationAcceptIn, - queryset: QuerySet = Depends(get_incoming_queryset), + queryset: InvitationQuerySet = Depends(get_incoming_queryset), ): invitation = get_object_or_404(queryset, uid=invitation_uid) @@ -201,7 +202,7 @@ def outgoing_create( with transaction.atomic(): try: - ret = models.CollectionInvitation.objects.create( + models.CollectionInvitation.objects.create( **data.dict(exclude={"collection", "username"}), user=to_user, fromMember=member ) except IntegrityError: @@ -212,7 +213,7 @@ def outgoing_create( def outgoing_list( iterator: t.Optional[str] = None, limit: int = 50, - queryset: QuerySet = Depends(get_outgoing_queryset), + queryset: InvitationQuerySet = Depends(get_outgoing_queryset), ): return list_common(queryset, iterator, limit) @@ -222,7 +223,7 @@ def outgoing_list( ) def outgoing_delete( invitation_uid: str, - queryset: QuerySet = Depends(get_outgoing_queryset), + queryset: InvitationQuerySet = Depends(get_outgoing_queryset), ): obj = get_object_or_404(queryset, uid=invitation_uid) obj.delete() diff --git a/etebase_fastapi/routers/member.py b/etebase_fastapi/routers/member.py index 210374c..41393bf 100644 --- a/etebase_fastapi/routers/member.py +++ b/etebase_fastapi/routers/member.py @@ -15,14 +15,15 @@ from .collection import get_collection, verify_collection_admin User = get_typed_user_model() member_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) -default_queryset: QuerySet = models.CollectionMember.objects.all() +MemberQuerySet = QuerySet[models.CollectionMember] +default_queryset: MemberQuerySet = models.CollectionMember.objects.all() -def get_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: +def get_queryset(collection: models.Collection = Depends(get_collection)) -> MemberQuerySet: return default_queryset.filter(collection=collection) -def get_member(username: str, queryset: QuerySet = Depends(get_queryset)) -> QuerySet: +def get_member(username: str, queryset: MemberQuerySet = Depends(get_queryset)) -> models.CollectionMember: return get_object_or_404(queryset, user__username__iexact=username) @@ -54,7 +55,7 @@ class MemberListResponse(BaseModel): def member_list( iterator: t.Optional[str] = None, limit: int = 50, - queryset: QuerySet = Depends(get_queryset), + queryset: MemberQuerySet = Depends(get_queryset), ): queryset = queryset.order_by("id") result, new_stoken_obj, done = filter_by_stoken_and_limit( diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 03f1a7d..c9db61c 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -7,7 +7,7 @@ import base64 from fastapi import status, Query, Depends from pydantic import BaseModel as PyBaseModel -from django.db.models import QuerySet +from django.db.models import Model, QuerySet from django.core.exceptions import ObjectDoesNotExist from django_etebase import app_settings @@ -22,6 +22,9 @@ Prefetch = Literal["auto", "medium"] PrefetchQuery = Query(default="auto") +T = t.TypeVar("T", bound=Model, covariant=True) + + class BaseModel(PyBaseModel): class Config: json_encoders = { @@ -35,7 +38,7 @@ class Context: prefetch: t.Optional[Prefetch] -def get_object_or_404(queryset: QuerySet, **kwargs): +def get_object_or_404(queryset: QuerySet[T], **kwargs) -> T: try: return queryset.get(**kwargs) except ObjectDoesNotExist as e: diff --git a/myauth/models.py b/myauth/models.py index c9298a4..89b94b4 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -15,7 +15,7 @@ class UnicodeUsernameValidator(validators.RegexValidator): class UserManager(DjangoUserManager): - def get_by_natural_key(self, username): + def get_by_natural_key(self, username: str): return self.get(**{self.model.USERNAME_FIELD + "__iexact": username}) @@ -37,7 +37,7 @@ class User(AbstractUser): ) @classmethod - def normalize_username(cls, username): + def normalize_username(cls, username: str): return super().normalize_username(username).lower() From c4235662d8a9b1f7b126f2f5162b9d6cefa815a3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 18:50:18 +0200 Subject: [PATCH 465/511] Use uvicorn[standard] in requirements to get some added extras. --- requirements.in/base.txt | 2 +- requirements.txt | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/requirements.in/base.txt b/requirements.in/base.txt index fee4a56..81af2a0 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -2,4 +2,4 @@ django msgpack pynacl fastapi -uvicorn +uvicorn[standard] diff --git a/requirements.txt b/requirements.txt index cfce456..22ed527 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,12 +10,18 @@ click==7.1.2 # via uvicorn django==3.1.4 # via -r requirements.in/base.txt fastapi==0.63.0 # via -r requirements.in/base.txt h11==0.11.0 # via uvicorn +httptools==0.1.1 # via uvicorn msgpack==1.0.2 # via -r requirements.in/base.txt pycparser==2.20 # via cffi pydantic==1.7.3 # via fastapi pynacl==1.4.0 # via -r requirements.in/base.txt +python-dotenv==0.15.0 # via uvicorn pytz==2020.4 # via django +pyyaml==5.3.1 # via uvicorn six==1.15.0 # via pynacl sqlparse==0.4.1 # via django starlette==0.13.6 # via fastapi -uvicorn==0.13.2 # via -r requirements.in/base.txt +uvicorn[standard]==0.13.2 # via -r requirements.in/base.txt +uvloop==0.14.0 # via uvicorn +watchgod==0.6 # via uvicorn +websockets==8.1 # via uvicorn From 8bfdbc55a3d8b9dfa5cdb2f38b87d1012c6b894c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 18:53:31 +0200 Subject: [PATCH 466/511] Add typing extensions to requirements. --- requirements.in/base.txt | 1 + requirements.txt | 1 + 2 files changed, 2 insertions(+) diff --git a/requirements.in/base.txt b/requirements.in/base.txt index 81af2a0..44e2875 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -2,4 +2,5 @@ django msgpack pynacl fastapi +typing_extensions uvicorn[standard] diff --git a/requirements.txt b/requirements.txt index 22ed527..f59431b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -21,6 +21,7 @@ pyyaml==5.3.1 # via uvicorn six==1.15.0 # via pynacl sqlparse==0.4.1 # via django starlette==0.13.6 # via fastapi +typing-extensions==3.7.4.3 # via -r requirements.in/base.txt uvicorn[standard]==0.13.2 # via -r requirements.in/base.txt uvloop==0.14.0 # via uvicorn watchgod==0.6 # via uvicorn From 981e1a9a6accf7913eb925126d4d9c827644360a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 21:00:56 +0200 Subject: [PATCH 467/511] Add mypy config. --- mypy.ini | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 mypy.ini diff --git a/mypy.ini b/mypy.ini new file mode 100644 index 0000000..0a96c4c --- /dev/null +++ b/mypy.ini @@ -0,0 +1,5 @@ +[mypy] +plugins = mypy_django_plugin.main + +[mypy.plugins.django-stubs] +django_settings_module = "etebase_server.settings" From 6615b149c5e334302ee21b999183d86b55b4c0c3 Mon Sep 17 00:00:00 2001 From: Simon Vandevelde Date: Tue, 29 Dec 2020 20:04:48 +0100 Subject: [PATCH 468/511] Update README with wiki changes (#82) * Changed link to wiki * Removed unneeded links (as uWSGI won't be supported) * Added --host flag to uvicorn command, as the server is only accessible from the host machine otherwise (and not from another in the local network) --- README.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1787a2f..c110672 100644 --- a/README.md +++ b/README.md @@ -62,18 +62,14 @@ Now you can initialise our django app. And you are done! You can now run the debug server just to see everything works as expected by running: ``` -uvicorn etebase_server.asgi:application --port 8000 +uvicorn etebase_server.asgi:application --host 0.0.0.0 --port 8000 ``` Using the debug server in production is not recommended, so please read the following section for a proper deployment. # Production deployment -There are more details about a proper production setup using Daphne and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-Daphne-and-Nginx). - -Etebase is based on Django so you should refer to one of the following - * The instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). - * Instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). +There are more details about a proper production setup using uvicorn and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-Nginx). The webserver should also be configured to serve Etebase using TLS. A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-Etebase) as well. From a7fdb4a1082b3d84a6ee7e7eae1086412124b183 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 29 Dec 2020 20:56:36 +0200 Subject: [PATCH 469/511] More typing fixes. --- django_etebase/models.py | 1 + etebase_fastapi/routers/collection.py | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/django_etebase/models.py b/django_etebase/models.py index 7725a19..fa56a95 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -56,6 +56,7 @@ class Collection(models.Model): @property def content(self) -> "CollectionItemRevision": + assert self.main_item is not None return self.main_item.content @property diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index 4825626..e20e955 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -107,6 +107,7 @@ class CollectionOut(CollectionCommon): def from_orm_context(cls: t.Type["CollectionOut"], obj: models.Collection, context: Context) -> "CollectionOut": member: models.CollectionMember = obj.members.get(user=context.user) collection_type = member.collectionType + assert obj.main_item is not None ret = cls( collectionType=collection_type and bytes(collection_type.uid), collectionKey=bytes(member.encryptionKey), @@ -299,8 +300,8 @@ def process_revisions_for_item(item: models.CollectionItem, revision_data: Colle revision.stoken = stoken revision.save() - for chunk in chunks_objs: - models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) + for chunk2 in chunks_objs: + models.RevisionChunkRelation.objects.create(chunk=chunk2, revision=revision) return revision @@ -383,6 +384,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val # We don't have to use select_for_update here because the unique constraint on current guards against # the race condition. But it's a good idea because it'll lock and wait rather than fail. current_revision = instance.revisions.filter(current=True).select_for_update().first() + assert current_revision is not None current_revision.current = None current_revision.save() @@ -523,8 +525,8 @@ def fetch_updates( new_stoken_obj = get_queryset_stoken(queryset) new_stoken = new_stoken_obj and new_stoken_obj.uid - stoken = stoken_rev and getattr(stoken_rev, "uid", None) - new_stoken = new_stoken or stoken + stoken_rev_uid = stoken_rev and getattr(stoken_rev, "uid", None) + new_stoken = new_stoken or stoken_rev_uid context = Context(user, prefetch) return CollectionItemListResponse( From 473448246f61a540110142f261e5aa2650b3cd12 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 30 Dec 2020 09:20:38 +0200 Subject: [PATCH 470/511] Add a django middleware to cleanup db connections. This severely impacts performance, though without it we are getting django.db.utils.InterfaceError once connections in the pool go stale. --- etebase_fastapi/main.py | 2 ++ etebase_fastapi/middleware.py | 15 +++++++++++++++ 2 files changed, 17 insertions(+) create mode 100644 etebase_fastapi/middleware.py diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index 8e8469c..c07d975 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -5,6 +5,7 @@ from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware +from .middleware import DjangoDbConnectionCleanupMiddleware from .exceptions import CustomHttpException from .msgpack import MsgpackResponse from .routers.authentication import authentication_router @@ -42,6 +43,7 @@ def create_application(prefix="", middlewares=[]): app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") + app.add_middleware(DjangoDbConnectionCleanupMiddleware) app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", diff --git a/etebase_fastapi/middleware.py b/etebase_fastapi/middleware.py new file mode 100644 index 0000000..06e347b --- /dev/null +++ b/etebase_fastapi/middleware.py @@ -0,0 +1,15 @@ +from starlette.types import ASGIApp, Receive, Scope, Send +from django.db import close_old_connections, reset_queries + + +class DjangoDbConnectionCleanupMiddleware: + def __init__(self, app: ASGIApp): + self.app = app + + async def __call__(self, scope: Scope, receive: Receive, send: Send): + reset_queries() + close_old_connections() + try: + await self.app(scope, receive, send) + finally: + close_old_connections() From 64be7f10bdf3dcddc00b46dbc4e27ea80ebbdd08 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 30 Dec 2020 10:17:01 +0200 Subject: [PATCH 471/511] Remove all of the needless async decorators. The code uses the django ORM which is sync, and fastapi handles sync paths just fine. So having all of this extra code for handling async was unnecessary. --- etebase_fastapi/routers/authentication.py | 20 +++++++++----------- etebase_fastapi/routers/collection.py | 18 ++++++++---------- 2 files changed, 17 insertions(+), 21 deletions(-) diff --git a/etebase_fastapi/routers/authentication.py b/etebase_fastapi/routers/authentication.py index efab7ef..be6a7e8 100644 --- a/etebase_fastapi/routers/authentication.py +++ b/etebase_fastapi/routers/authentication.py @@ -7,7 +7,6 @@ import nacl.encoding import nacl.hash import nacl.secret import nacl.signing -from asgiref.sync import sync_to_async from django.conf import settings from django.contrib.auth import user_logged_out, user_logged_in from django.core import exceptions as django_exceptions @@ -145,7 +144,6 @@ def save_changed_password(data: ChangePassword, user: UserType): user_info.save() -@sync_to_async def validate_login_request( validated_data: LoginResponse, challenge_sent_to_user: Authentication, @@ -193,13 +191,13 @@ def login_challenge(user: UserType = Depends(get_login_user)): @authentication_router.post("/login/", response_model=LoginOut) -async def login(data: Login, request: Request): - user = await sync_to_async(get_login_user)(request, LoginChallengeIn(username=data.response_data.username)) +def login(data: Login, request: Request): + user = get_login_user(request, LoginChallengeIn(username=data.response_data.username)) host = request.headers.get("Host") - await validate_login_request(data.response_data, data, user, "login", host) - data = await sync_to_async(LoginOut.from_orm)(user) - await sync_to_async(user_logged_in.send)(sender=user.__class__, request=None, user=user) - return data + validate_login_request(data.response_data, data, user, "login", host) + ret = LoginOut.from_orm(user) + user_logged_in.send(sender=user.__class__, request=None, user=user) + return ret @authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) @@ -209,10 +207,10 @@ def logout(auth_data: AuthData = Depends(get_auth_data)): @authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) -async def change_password(data: ChangePassword, request: Request, user: UserType = Depends(get_authenticated_user)): +def change_password(data: ChangePassword, request: Request, user: UserType = Depends(get_authenticated_user)): host = request.headers.get("Host") - await validate_login_request(data.response_data, data, user, "changePassword", host) - await sync_to_async(save_changed_password)(data, user) + validate_login_request(data.response_data, data, user, "changePassword", host) + save_changed_password(data, user) @authentication_router.post("/dashboard_url/", responses=permission_responses) diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index e20e955..b716105 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -188,7 +188,6 @@ class ItemBatchIn(BaseModel): ) -@sync_to_async def collection_list_common( queryset: CollectionQuerySet, user: UserType, @@ -248,7 +247,7 @@ def has_write_access( response_model_exclude_unset=True, dependencies=PERMISSIONS_READ, ) -async def list_multi( +def list_multi( data: ListMulti, stoken: t.Optional[str] = None, limit: int = 50, @@ -261,18 +260,18 @@ async def list_multi( Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) ) - return await collection_list_common(queryset, user, stoken, limit, prefetch) + return collection_list_common(queryset, user, stoken, limit, prefetch) @collection_router.get("/", response_model=CollectionListResponse, dependencies=PERMISSIONS_READ) -async def collection_list( +def collection_list( stoken: t.Optional[str] = None, limit: int = 50, prefetch: Prefetch = PrefetchQuery, user: UserType = Depends(get_authenticated_user), queryset: CollectionQuerySet = Depends(get_collection_queryset), ): - return await collection_list_common(queryset, user, stoken, limit, prefetch) + return collection_list_common(queryset, user, stoken, limit, prefetch) def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionInOut): @@ -341,8 +340,8 @@ def _create(data: CollectionIn, user: UserType): @collection_router.post("/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE) -async def create(data: CollectionIn, user: UserType = Depends(get_authenticated_user)): - await sync_to_async(_create)(data, user) +def create(data: CollectionIn, user: UserType = Depends(get_authenticated_user)): + _create(data, user) @collection_router.get("/{collection_uid}/", response_model=CollectionOut, dependencies=PERMISSIONS_READ) @@ -407,7 +406,6 @@ def item_get( return CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) -@sync_to_async def item_list_common( queryset: CollectionItemQuerySet, user: UserType, @@ -425,7 +423,7 @@ def item_list_common( @item_router.get("/item/", response_model=CollectionItemListResponse, dependencies=PERMISSIONS_READ) -async def item_list( +def item_list( queryset: CollectionItemQuerySet = Depends(get_item_queryset), stoken: t.Optional[str] = None, limit: int = 50, @@ -436,7 +434,7 @@ async def item_list( if not withCollection: queryset = queryset.filter(parent__isnull=True) - response = await item_list_common(queryset, user, stoken, limit, prefetch) + response = item_list_common(queryset, user, stoken, limit, prefetch) return response From 6738c2cf20bb1d5a0e24c9910bb0345fb59716cf Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 30 Dec 2020 13:55:05 +0200 Subject: [PATCH 472/511] Remove unused variable. --- etebase_fastapi/routers/collection.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index b716105..2041ce7 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -8,7 +8,7 @@ from django.db.models import Q, QuerySet from fastapi import APIRouter, Depends, status, Request from django_etebase import models -from myauth.models import UserType, get_typed_user_model +from myauth.models import UserType from .authentication import get_authenticated_user from ..exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError from ..msgpack import MsgpackRoute @@ -27,7 +27,6 @@ from ..utils import ( from ..dependencies import get_collection_queryset, get_item_queryset, get_collection from ..sendfile import sendfile -User = get_typed_user_model collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) CollectionQuerySet = QuerySet[models.Collection] From 6ec03c3d3494f91a59accfced180b03eb02478a1 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 30 Dec 2020 15:37:59 +0200 Subject: [PATCH 473/511] Revert "Add a django middleware to cleanup db connections." This ended up being useless because of the way startlette and fastapi do thread pools. The middleware is called in one thread, the path in another, and the dependency in yet another. This reverts commit 473448246f61a540110142f261e5aa2650b3cd12. --- etebase_fastapi/main.py | 2 -- etebase_fastapi/middleware.py | 15 --------------- 2 files changed, 17 deletions(-) delete mode 100644 etebase_fastapi/middleware.py diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index c07d975..8e8469c 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -5,7 +5,6 @@ from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware -from .middleware import DjangoDbConnectionCleanupMiddleware from .exceptions import CustomHttpException from .msgpack import MsgpackResponse from .routers.authentication import authentication_router @@ -43,7 +42,6 @@ def create_application(prefix="", middlewares=[]): app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") - app.add_middleware(DjangoDbConnectionCleanupMiddleware) app.add_middleware( CORSMiddleware, allow_origin_regex="https?://.*", diff --git a/etebase_fastapi/middleware.py b/etebase_fastapi/middleware.py deleted file mode 100644 index 06e347b..0000000 --- a/etebase_fastapi/middleware.py +++ /dev/null @@ -1,15 +0,0 @@ -from starlette.types import ASGIApp, Receive, Scope, Send -from django.db import close_old_connections, reset_queries - - -class DjangoDbConnectionCleanupMiddleware: - def __init__(self, app: ASGIApp): - self.app = app - - async def __call__(self, scope: Scope, receive: Receive, send: Send): - reset_queries() - close_old_connections() - try: - await self.app(scope, receive, send) - finally: - close_old_connections() From 5b8f667e55cef4ea5ed1e2ad78f6df7d7101258c Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 30 Dec 2020 17:09:16 +0200 Subject: [PATCH 474/511] Cleanup django db connections before every request and every dependency. This is instead of the commit we reverted in the previous commit. The problem is that django keeps the connection per thread and it relies on django itself to clean them up before/after connections. We can't do this, because django is unaware of fastapi, so we have to manage this ourselves. The easiest way is to call it at the beginning of evenry route and every dep. We need to do it for each because unfortunately fastapi may send them to different worker threads. --- etebase_fastapi/db_hack.py | 27 +++++++++++++++++++++++++++ etebase_fastapi/msgpack.py | 19 +++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 etebase_fastapi/db_hack.py diff --git a/etebase_fastapi/db_hack.py b/etebase_fastapi/db_hack.py new file mode 100644 index 0000000..24d5824 --- /dev/null +++ b/etebase_fastapi/db_hack.py @@ -0,0 +1,27 @@ +""" +FIXME: this whole function is a hack around the django db limitations due to how db connections are cached and cleaned. +Essentially django assumes there's the django request dispatcher to automatically clean up after the ORM. +""" +import typing as t +from functools import wraps + +from django.db import close_old_connections, reset_queries + + +def django_db_cleanup(): + reset_queries() + close_old_connections() + + +def django_db_cleanup_decorator(func: t.Callable[..., t.Any]): + from inspect import iscoroutinefunction + + if iscoroutinefunction(func): + return func + + @wraps(func) + def wrapper(*args, **kwargs): + django_db_cleanup() + return func(*args, **kwargs) + + return wrapper diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index 8de8806..67627e1 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -1,10 +1,13 @@ import typing as t + +from fastapi import params from fastapi.routing import APIRoute, get_request_handler from pydantic import BaseModel from starlette.requests import Request from starlette.responses import Response from .utils import msgpack_encode, msgpack_decode +from .db_hack import django_db_cleanup_decorator class MsgpackRequest(Request): @@ -35,6 +38,22 @@ class MsgpackRoute(APIRoute): # keep track of content-type -> response classes ROUTES_HANDLERS_CLASSES = {MsgpackResponse.media_type: MsgpackResponse} + def __init__( + self, + path: str, + endpoint: t.Callable[..., t.Any], + *args, + dependencies: t.Optional[t.Sequence[params.Depends]] = None, + **kwargs + ): + if dependencies is not None: + dependencies = [ + params.Depends(django_db_cleanup_decorator(dep.dependency), use_cache=dep.use_cache) + for dep in dependencies + ] + endpoint = django_db_cleanup_decorator(endpoint) + super().__init__(path, endpoint, *args, dependencies=dependencies, **kwargs) + def _get_media_type_route_handler(self, media_type): return get_request_handler( dependant=self.dependant, From 2e21fe4994c2bf9799b36e535db8bdf6943ff957 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 31 Dec 2020 10:03:16 +0200 Subject: [PATCH 475/511] Django db cleanup: explicitly add it to dependencies. We can't really add it manually, because some of the deps are auto included as parameters. These were not being decorated which in turn meeant issues. --- etebase_fastapi/dependencies.py | 6 ++++++ etebase_fastapi/msgpack.py | 17 ++--------------- etebase_fastapi/routers/collection.py | 3 +++ etebase_fastapi/routers/invitation.py | 3 +++ etebase_fastapi/routers/member.py | 3 +++ 5 files changed, 17 insertions(+), 15 deletions(-) diff --git a/etebase_fastapi/dependencies.py b/etebase_fastapi/dependencies.py index fb9cec5..520d499 100644 --- a/etebase_fastapi/dependencies.py +++ b/etebase_fastapi/dependencies.py @@ -11,6 +11,7 @@ from django_etebase.token_auth.models import AuthToken, get_default_expiry from myauth.models import UserType, get_typed_user_model from .exceptions import AuthenticationFailed from .utils import get_object_or_404 +from .db_hack import django_db_cleanup_decorator User = get_typed_user_model() @@ -55,25 +56,30 @@ def __get_authenticated_user(api_token: str): return token.user, token +@django_db_cleanup_decorator def get_auth_data(api_token: str = Depends(token_scheme)) -> AuthData: user, token = __get_authenticated_user(api_token) return AuthData(user, token) +@django_db_cleanup_decorator def get_authenticated_user(api_token: str = Depends(token_scheme)) -> UserType: user, _ = __get_authenticated_user(api_token) return user +@django_db_cleanup_decorator def get_collection_queryset(user: UserType = Depends(get_authenticated_user)) -> QuerySet: default_queryset: QuerySet = models.Collection.objects.all() return default_queryset.filter(members__user=user) +@django_db_cleanup_decorator def get_collection(collection_uid: str, queryset: QuerySet = Depends(get_collection_queryset)) -> models.Collection: return get_object_or_404(queryset, uid=collection_uid) +@django_db_cleanup_decorator def get_item_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: default_item_queryset: QuerySet = models.CollectionItem.objects.all() # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index 67627e1..a671e79 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -1,6 +1,5 @@ import typing as t -from fastapi import params from fastapi.routing import APIRoute, get_request_handler from pydantic import BaseModel from starlette.requests import Request @@ -38,21 +37,9 @@ class MsgpackRoute(APIRoute): # keep track of content-type -> response classes ROUTES_HANDLERS_CLASSES = {MsgpackResponse.media_type: MsgpackResponse} - def __init__( - self, - path: str, - endpoint: t.Callable[..., t.Any], - *args, - dependencies: t.Optional[t.Sequence[params.Depends]] = None, - **kwargs - ): - if dependencies is not None: - dependencies = [ - params.Depends(django_db_cleanup_decorator(dep.dependency), use_cache=dep.use_cache) - for dep in dependencies - ] + def __init__(self, path: str, endpoint: t.Callable[..., t.Any], *args, **kwargs): endpoint = django_db_cleanup_decorator(endpoint) - super().__init__(path, endpoint, *args, dependencies=dependencies, **kwargs) + super().__init__(path, endpoint, *args, **kwargs) def _get_media_type_route_handler(self, media_type): return get_request_handler( diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index 2041ce7..4dcb3c6 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -26,6 +26,7 @@ from ..utils import ( ) from ..dependencies import get_collection_queryset, get_item_queryset, get_collection from ..sendfile import sendfile +from ..db_hack import django_db_cleanup_decorator collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -222,6 +223,7 @@ def collection_list_common( # permissions +@django_db_cleanup_decorator def verify_collection_admin( collection: models.Collection = Depends(get_collection), user: UserType = Depends(get_authenticated_user) ): @@ -229,6 +231,7 @@ def verify_collection_admin( raise PermissionDenied("admin_access_required", "Only collection admins can perform this operation.") +@django_db_cleanup_decorator def has_write_access( collection: models.Collection = Depends(get_collection), user: UserType = Depends(get_authenticated_user) ): diff --git a/etebase_fastapi/routers/invitation.py b/etebase_fastapi/routers/invitation.py index aceb05d..cbe570b 100644 --- a/etebase_fastapi/routers/invitation.py +++ b/etebase_fastapi/routers/invitation.py @@ -19,6 +19,7 @@ from ..utils import ( PERMISSIONS_READ, PERMISSIONS_READWRITE, ) +from ..db_hack import django_db_cleanup_decorator User = get_typed_user_model() invitation_incoming_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -86,10 +87,12 @@ class InvitationListResponse(BaseModel): done: bool +@django_db_cleanup_decorator def get_incoming_queryset(user: UserType = Depends(get_authenticated_user)): return default_queryset.filter(user=user) +@django_db_cleanup_decorator def get_outgoing_queryset(user: UserType = Depends(get_authenticated_user)): return default_queryset.filter(fromMember__user=user) diff --git a/etebase_fastapi/routers/member.py b/etebase_fastapi/routers/member.py index 41393bf..38beb79 100644 --- a/etebase_fastapi/routers/member.py +++ b/etebase_fastapi/routers/member.py @@ -10,6 +10,7 @@ from .authentication import get_authenticated_user from ..msgpack import MsgpackRoute from ..utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE from ..stoken_handler import filter_by_stoken_and_limit +from ..db_hack import django_db_cleanup_decorator from .collection import get_collection, verify_collection_admin @@ -19,10 +20,12 @@ MemberQuerySet = QuerySet[models.CollectionMember] default_queryset: MemberQuerySet = models.CollectionMember.objects.all() +@django_db_cleanup_decorator def get_queryset(collection: models.Collection = Depends(get_collection)) -> MemberQuerySet: return default_queryset.filter(collection=collection) +@django_db_cleanup_decorator def get_member(username: str, queryset: MemberQuerySet = Depends(get_queryset)) -> models.CollectionMember: return get_object_or_404(queryset, user__username__iexact=username) From 84870d25bfc1a3529e3349ebfccb62d806dfb154 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 3 Jan 2021 09:14:10 +0200 Subject: [PATCH 476/511] README: mention that Windows requires WSL --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c110672..f8800ff 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Before installing the Etebase server make sure you install `virtualenv` (for **P * Arch Linux: `pacman -S python-virtualenv` * Debian/Ubuntu: `apt-get install python3-virtualenv` -* Mac/Windows/Other Linux: install virtualenv or just skip the instructions mentioning virtualenv. +* Mac/Windows (WSL)/Other Linux: install virtualenv or just skip the instructions mentioning virtualenv. Then just clone the git repo and set up this app: From 3894fd205b54302d113898ae73a60366807b8b31 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 3 Jan 2021 12:55:24 +0200 Subject: [PATCH 477/511] README: mention minimum requirements --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index f8800ff..1d39872 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,10 @@ An [Etebase](https://www.etebase.com) (EteSync 2.0) server so you can run your o # Installation +## Requirements + +Etebase requires Python 3.7 or newer and has a few Python dependencies (listed in `requirements.in/base.txt`). + ## From source Before installing the Etebase server make sure you install `virtualenv` (for **Python 3**): From 43dede57d298bc3d1e0943ba0228a097ac0b019b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 4 Jan 2021 10:13:24 +0200 Subject: [PATCH 478/511] Exceptions: fix types. --- etebase_fastapi/exceptions.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index 72a3faf..d85c07e 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -2,6 +2,7 @@ from fastapi import status import typing as t from pydantic import BaseModel +from django.core.exceptions import ValidationError as DjangoValidationError class HttpErrorField(BaseModel): @@ -92,8 +93,8 @@ class ValidationError(HttpError): super().__init__(code=code, detail=detail, errors=errors, status_code=status_code) -def flatten_errors(field_name, errors) -> t.List[HttpError]: - ret = [] +def flatten_errors(field_name: str, errors) -> t.List[HttpError]: + ret: t.List[HttpError] = [] if isinstance(errors, dict): for error_key in errors: error = errors[error_key] @@ -104,11 +105,11 @@ def flatten_errors(field_name, errors) -> t.List[HttpError]: message = error.messages[0] else: message = str(error) - ret.append(dict(code=error.code, detail=message, field=field_name)) + ret.append(ValidationError(code=error.code, detail=message, field=field_name)) return ret -def transform_validation_error(prefix, err): +def transform_validation_error(prefix: str, err: DjangoValidationError): if hasattr(err, "error_dict"): errors = flatten_errors(prefix, err.error_dict) elif not hasattr(err, "message"): From 9bf118225d51c25371c3e9f572f1de7ae4c5ddf7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 4 Jan 2021 10:14:56 +0200 Subject: [PATCH 479/511] Exceptions: fix error when transforming django validation errors. --- etebase_fastapi/exceptions.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index d85c07e..d38ef42 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -115,5 +115,5 @@ def transform_validation_error(prefix: str, err: DjangoValidationError): elif not hasattr(err, "message"): errors = flatten_errors(prefix, err.error_list) else: - raise HttpError(err.code, err.message) + raise HttpError(err.code or "validation_error", err.message) raise HttpError(code="field_errors", detail="Field validations failed.", errors=errors) From 9f0430a6da8a1de6916edba0b41411e9b0d68cc5 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 4 Jan 2021 11:56:17 +0200 Subject: [PATCH 480/511] Improve types. --- etebase_fastapi/routers/authentication.py | 2 +- etebase_fastapi/utils.py | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/etebase_fastapi/routers/authentication.py b/etebase_fastapi/routers/authentication.py index be6a7e8..f8628b2 100644 --- a/etebase_fastapi/routers/authentication.py +++ b/etebase_fastapi/routers/authentication.py @@ -125,7 +125,7 @@ def get_login_user(request: Request, challenge: LoginChallengeIn) -> UserType: raise AuthenticationFailed(code="user_not_found", detail="User not found") -def get_encryption_key(salt): +def get_encryption_key(salt: bytes): key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) return nacl.hash.blake2b( b"", diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index c9db61c..9f915e2 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -50,21 +50,21 @@ def is_collection_admin(collection, user): return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) -def msgpack_encode(content): +def msgpack_encode(content) -> bytes: ret = msgpack.packb(content, use_bin_type=True) assert ret is not None return ret -def msgpack_decode(content): +def msgpack_decode(content: bytes): return msgpack.unpackb(content, raw=False) -def b64encode(value): +def b64encode(value: bytes): return base64.urlsafe_b64encode(value).decode("ascii").strip("=") -def b64decode(data): +def b64decode(data: str): data += "=" * ((4 - len(data) % 4) % 4) return base64.urlsafe_b64decode(data) From 1349f99cd39df34df87b0a53aecf8117876fd0a2 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 4 Jan 2021 16:31:28 +0200 Subject: [PATCH 481/511] Exceptions: inherit from the fastapi HTTPException. --- etebase_fastapi/exceptions.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index d38ef42..61e4439 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -1,4 +1,4 @@ -from fastapi import status +from fastapi import status, HTTPException import typing as t from pydantic import BaseModel @@ -23,11 +23,10 @@ class HttpErrorOut(BaseModel): orm_mode = True -class CustomHttpException(Exception): +class CustomHttpException(HTTPException): def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): - self.status_code = status_code self.code = code - self.detail = detail + super().__init__(status_code, detail) @property def as_dict(self) -> dict: From cd4131e890bc9abe8bbedab726b8bed5acb9ef4e Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 11 Jan 2021 22:12:31 +0200 Subject: [PATCH 482/511] Exceptions: make sure error codes aren't None. --- etebase_fastapi/exceptions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index 61e4439..d38d50a 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -72,7 +72,7 @@ class HttpError(CustomHttpException): errors: t.Optional[t.List["HttpError"]] = None, ): self.errors = errors - super().__init__(code=code, detail=detail, status_code=status_code) + super().__init__(code=code or "generic_error", detail=detail, status_code=status_code) @property def as_dict(self) -> dict: @@ -104,7 +104,7 @@ def flatten_errors(field_name: str, errors) -> t.List[HttpError]: message = error.messages[0] else: message = str(error) - ret.append(ValidationError(code=error.code, detail=message, field=field_name)) + ret.append(ValidationError(code=error.code or "validation_error", detail=message, field=field_name)) return ret From f52facad1c0adaa84b9418428b35c8e4fb067bf8 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 4 Jan 2021 10:02:47 +0200 Subject: [PATCH 483/511] Subscriptions: implement live subscriptions for collection items --- django_etebase/app_settings_inner.py | 6 ++ etebase_fastapi/exceptions.py | 10 +++ etebase_fastapi/main.py | 16 ++++ etebase_fastapi/redis.py | 27 ++++++ etebase_fastapi/routers/collection.py | 26 +++++- etebase_fastapi/routers/websocket.py | 114 ++++++++++++++++++++++++++ 6 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 etebase_fastapi/redis.py create mode 100644 etebase_fastapi/routers/websocket.py diff --git a/django_etebase/app_settings_inner.py b/django_etebase/app_settings_inner.py index 90225a6..41fd910 100644 --- a/django_etebase/app_settings_inner.py +++ b/django_etebase/app_settings_inner.py @@ -11,6 +11,8 @@ # # You should have received a copy of the GNU General Public License # along with this program. If not, see . +import typing as t + from django.utils.functional import cached_property @@ -31,6 +33,10 @@ class AppSettings: return getattr(settings, self.prefix + name, dflt) + @cached_property + def REDIS_URI(self) -> t.Optional[str]: # pylint: disable=invalid-name + return self._setting("REDIS_URI", None) + @cached_property def API_PERMISSIONS_READ(self): # pylint: disable=invalid-name perms = self._setting("API_PERMISSIONS_READ", tuple()) diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py index d38d50a..1a98fcb 100644 --- a/etebase_fastapi/exceptions.py +++ b/etebase_fastapi/exceptions.py @@ -63,6 +63,16 @@ class PermissionDenied(CustomHttpException): super().__init__(code=code, detail=detail, status_code=status_code) +class NotSupported(CustomHttpException): + def __init__( + self, + code="not_implemented", + detail: str = "This server's configuration does not support this request.", + status_code: int = status.HTTP_501_NOT_IMPLEMENTED, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + class HttpError(CustomHttpException): def __init__( self, diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index 8e8469c..d63c01d 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -5,12 +5,15 @@ from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware +from django_etebase import app_settings + from .exceptions import CustomHttpException from .msgpack import MsgpackResponse from .routers.authentication import authentication_router from .routers.collection import collection_router, item_router from .routers.member import member_router from .routers.invitation import invitation_incoming_router, invitation_outgoing_router +from .routers.websocket import websocket_router def create_application(prefix="", middlewares=[]): @@ -36,6 +39,7 @@ def create_application(prefix="", middlewares=[]): app.include_router( invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"] ) + app.include_router(websocket_router, prefix=f"{BASE_PATH}/ws", tags=["websocket"]) if settings.DEBUG: from etebase_fastapi.routers.test_reset_view import test_reset_view_router @@ -54,6 +58,18 @@ def create_application(prefix="", middlewares=[]): for middleware in middlewares: app.add_middleware(middleware) + @app.on_event("startup") + async def on_startup() -> None: + from .redis import redisw + + await redisw.setup() + + @app.on_event("shutdown") + async def on_shutdown(): + from .redis import redisw + + await redisw.close() + @app.exception_handler(CustomHttpException) async def custom_exception_handler(request: Request, exc: CustomHttpException): return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) diff --git a/etebase_fastapi/redis.py b/etebase_fastapi/redis.py new file mode 100644 index 0000000..3735e36 --- /dev/null +++ b/etebase_fastapi/redis.py @@ -0,0 +1,27 @@ +import typing as t +import aioredis + +from django_etebase import app_settings + + +class RedisWrapper: + redis: aioredis.Redis + + def __init__(self, redis_uri: t.Optional[str]): + self.redis_uri = redis_uri + + async def setup(self): + if self.redis_uri is not None: + self.redis = await aioredis.create_redis_pool(self.redis_uri) + + async def close(self): + if self.redis is not None: + self.redis.close() + await self.redis.wait_closed() + + @property + def is_active(self): + return self.redis_uri is not None + + +redisw = RedisWrapper(app_settings.REDIS_URI) diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index 4dcb3c6..df25541 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -1,6 +1,6 @@ import typing as t -from asgiref.sync import sync_to_async +from asgiref.sync import sync_to_async, async_to_sync from django.core import exceptions as django_exceptions from django.core.files.base import ContentFile from django.db import transaction, IntegrityError @@ -10,6 +10,7 @@ from fastapi import APIRouter, Depends, status, Request from django_etebase import models from myauth.models import UserType from .authentication import get_authenticated_user +from .websocket import get_ticket, TicketRequest, TicketOut from ..exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError from ..msgpack import MsgpackRoute from ..stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken @@ -19,6 +20,7 @@ from ..utils import ( Prefetch, PrefetchQuery, is_collection_admin, + msgpack_encode, BaseModel, permission_responses, PERMISSIONS_READ, @@ -26,6 +28,7 @@ from ..utils import ( ) from ..dependencies import get_collection_queryset, get_item_queryset, get_collection from ..sendfile import sendfile +from ..redis import redisw from ..db_hack import django_db_cleanup_decorator collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) @@ -188,6 +191,16 @@ class ItemBatchIn(BaseModel): ) +# FIXME: make it a background task +def report_items_changed(col_uid: str, stoken: str, items: t.List[CollectionItemIn]): + if not redisw.is_active: + return + + redis = redisw.redis + content = msgpack_encode(CollectionItemListResponse(data=items, stoken=stoken, done=True).dict()) + async_to_sync(redis.publish)(f"col.{col_uid}", content) + + def collection_list_common( queryset: CollectionQuerySet, user: UserType, @@ -440,6 +453,15 @@ def item_list( return response +@item_router.post("/item/subscription-ticket/", response_model=TicketOut, dependencies=PERMISSIONS_READ) +async def item_list_subscription_ticket( + collection: models.Collection = Depends(get_collection), + user: UserType = Depends(get_authenticated_user), +): + """Get an authentication ticket that can be used with the websocket endpoint""" + return await get_ticket(TicketRequest(collection=collection.uid), user) + + def item_bulk_common(data: ItemBatchIn, user: UserType, stoken: t.Optional[str], uid: str, validate_etag: bool): queryset = get_collection_queryset(user) with transaction.atomic(): # We need this for locking the collection object @@ -465,6 +487,8 @@ def item_bulk_common(data: ItemBatchIn, user: UserType, stoken: t.Optional[str], status_code=status.HTTP_409_CONFLICT, ) + report_items_changed(collection_object.uid, collection_object.stoken, data.items) + @item_router.get( "/item/{item_uid}/revision/", response_model=CollectionItemRevisionListResponse, dependencies=PERMISSIONS_READ diff --git a/etebase_fastapi/routers/websocket.py b/etebase_fastapi/routers/websocket.py new file mode 100644 index 0000000..2d599db --- /dev/null +++ b/etebase_fastapi/routers/websocket.py @@ -0,0 +1,114 @@ +import asyncio +import typing as t + +import aioredis +from django.db.models import QuerySet +from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect, status +import nacl.encoding +import nacl.utils + +from django_etebase import models +from django_etebase.utils import CallbackContext, get_user_queryset +from myauth.models import UserType, get_typed_user_model + +from ..exceptions import NotSupported +from ..msgpack import MsgpackRoute, msgpack_decode, msgpack_encode +from ..redis import redisw +from ..utils import BaseModel, permission_responses + + +User = get_typed_user_model() +websocket_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +CollectionQuerySet = QuerySet[models.Collection] + + +TICKET_VALIDITY_SECONDS = 10 + + +class TicketRequest(BaseModel): + collection: str + + +class TicketOut(BaseModel): + ticket: str + + +class TicketInner(BaseModel): + user: int + req: TicketRequest + + +async def get_ticket( + ticket_request: TicketRequest, + user: UserType, +): + """Get an authentication ticket that can be used with the websocket endpoint for authentication""" + if not redisw.is_active: + raise NotSupported(detail="This end-point requires Redis to be configured") + + uid = nacl.encoding.URLSafeBase64Encoder.encode(nacl.utils.random(32)) + ticket_model = TicketInner(user=user.id, req=ticket_request) + ticket_raw = msgpack_encode(ticket_model.dict()) + await redisw.redis.set(uid, ticket_raw, expire=TICKET_VALIDITY_SECONDS * 1000) + return TicketOut(ticket=uid) + + +async def load_websocket_ticket(websocket: WebSocket, ticket: str) -> t.Optional[TicketInner]: + content = await redisw.redis.get(ticket) + if content is None: + await websocket.close(code=status.WS_1008_POLICY_VIOLATION) + return None + await redisw.redis.delete(ticket) + return TicketInner(**msgpack_decode(content)) + + +def get_websocket_user(websocket: WebSocket, ticket_model: t.Optional[TicketInner] = Depends(load_websocket_ticket)): + if ticket_model is None: + return None + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(websocket.path_params)) + return user_queryset.get(id=ticket_model.user) + + +@websocket_router.websocket("/{ticket}/") +async def websocket_endpoint( + websocket: WebSocket, + user: t.Optional[UserType] = Depends(get_websocket_user), + ticket_model: TicketInner = Depends(load_websocket_ticket), +): + if user is None: + return + await websocket.accept() + await redis_connector(websocket, ticket_model) + + +async def redis_connector(websocket: WebSocket, ticket_model: TicketInner): + async def producer_handler(r: aioredis.Redis, ws: WebSocket): + channel_name = f"col.{ticket_model.req.collection}" + (channel,) = await r.psubscribe(channel_name) + assert isinstance(channel, aioredis.Channel) + try: + while True: + # We wait on the websocket so we fail if web sockets fail or get data + receive = asyncio.create_task(websocket.receive()) + done, pending = await asyncio.wait( + {receive, channel.wait_message()}, return_when=asyncio.FIRST_COMPLETED + ) + for task in pending: + task.cancel() + if receive in done: + # Web socket should never receieve any data + await websocket.close(code=status.WS_1008_POLICY_VIOLATION) + return + + message_raw = t.cast(t.Optional[t.Tuple[str, bytes]], await channel.get()) + if message_raw: + _, message = message_raw + await ws.send_bytes(message) + + except aioredis.errors.ConnectionClosedError: + await websocket.close(code=status.WS_1012_SERVICE_RESTART) + except WebSocketDisconnect: + pass + + redis = redisw.redis + await producer_handler(redis, websocket) From 61bd82f1e3dc6a993d823b0941fbb6f7a864f3fa Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 11 Jan 2021 18:39:01 +0200 Subject: [PATCH 484/511] Subscriptions: stream missing items if user passed an old stoken. --- etebase_fastapi/routers/websocket.py | 35 ++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/routers/websocket.py b/etebase_fastapi/routers/websocket.py index 2d599db..ad3331b 100644 --- a/etebase_fastapi/routers/websocket.py +++ b/etebase_fastapi/routers/websocket.py @@ -2,6 +2,7 @@ import asyncio import typing as t import aioredis +from asgiref.sync import sync_to_async from django.db.models import QuerySet from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect, status import nacl.encoding @@ -11,6 +12,7 @@ from django_etebase import models from django_etebase.utils import CallbackContext, get_user_queryset from myauth.models import UserType, get_typed_user_model +from ..dependencies import get_collection_queryset, get_item_queryset from ..exceptions import NotSupported from ..msgpack import MsgpackRoute, msgpack_decode, msgpack_encode from ..redis import redisw @@ -72,20 +74,49 @@ def get_websocket_user(websocket: WebSocket, ticket_model: t.Optional[TicketInne @websocket_router.websocket("/{ticket}/") async def websocket_endpoint( websocket: WebSocket, + stoken: t.Optional[str], user: t.Optional[UserType] = Depends(get_websocket_user), ticket_model: TicketInner = Depends(load_websocket_ticket), ): if user is None: return await websocket.accept() - await redis_connector(websocket, ticket_model) + await redis_connector(websocket, ticket_model, user, stoken) -async def redis_connector(websocket: WebSocket, ticket_model: TicketInner): +async def send_item_updates( + websocket: WebSocket, + collection: models.Collection, + user: UserType, + stoken: t.Optional[str], +): + from .collection import item_list_common + + done = False + while not done: + queryset = await sync_to_async(get_item_queryset)(collection) + response = await sync_to_async(item_list_common)(queryset, user, stoken, limit=50, prefetch="auto") + done = response.done + if len(response.data) > 0: + await websocket.send_bytes(msgpack_encode(response.dict())) + + +async def redis_connector(websocket: WebSocket, ticket_model: TicketInner, user: UserType, stoken: t.Optional[str]): async def producer_handler(r: aioredis.Redis, ws: WebSocket): channel_name = f"col.{ticket_model.req.collection}" (channel,) = await r.psubscribe(channel_name) assert isinstance(channel, aioredis.Channel) + + # Send missing items if we are not up to date + queryset: QuerySet[models.Collection] = get_collection_queryset(user) + collection: t.Optional[models.Collection] = await sync_to_async( + queryset.filter(uid=ticket_model.req.collection).first + )() + if collection is None: + await websocket.close(code=status.WS_1008_POLICY_VIOLATION) + return + await send_item_updates(websocket, collection, user, stoken) + try: while True: # We wait on the websocket so we fail if web sockets fail or get data From 240469342bd57a32fdd54d0f106ce01dc75ae6c6 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Tue, 12 Jan 2021 11:57:43 +0200 Subject: [PATCH 485/511] Move reporting item changes to a background task. --- etebase_fastapi/routers/collection.py | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index df25541..167179d 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -1,11 +1,11 @@ import typing as t -from asgiref.sync import sync_to_async, async_to_sync +from asgiref.sync import sync_to_async from django.core import exceptions as django_exceptions from django.core.files.base import ContentFile from django.db import transaction, IntegrityError from django.db.models import Q, QuerySet -from fastapi import APIRouter, Depends, status, Request +from fastapi import APIRouter, Depends, status, Request, BackgroundTasks from django_etebase import models from myauth.models import UserType @@ -191,14 +191,13 @@ class ItemBatchIn(BaseModel): ) -# FIXME: make it a background task -def report_items_changed(col_uid: str, stoken: str, items: t.List[CollectionItemIn]): +async def report_items_changed(col_uid: str, stoken: str, items: t.List[CollectionItemIn]): if not redisw.is_active: return redis = redisw.redis content = msgpack_encode(CollectionItemListResponse(data=items, stoken=stoken, done=True).dict()) - async_to_sync(redis.publish)(f"col.{col_uid}", content) + await redis.publish(f"col.{col_uid}", content) def collection_list_common( @@ -462,7 +461,14 @@ async def item_list_subscription_ticket( return await get_ticket(TicketRequest(collection=collection.uid), user) -def item_bulk_common(data: ItemBatchIn, user: UserType, stoken: t.Optional[str], uid: str, validate_etag: bool): +def item_bulk_common( + data: ItemBatchIn, + user: UserType, + stoken: t.Optional[str], + uid: str, + validate_etag: bool, + background_tasks: BackgroundTasks, +): queryset = get_collection_queryset(user) with transaction.atomic(): # We need this for locking the collection object collection_object = queryset.select_for_update().get(uid=uid) @@ -487,7 +493,7 @@ def item_bulk_common(data: ItemBatchIn, user: UserType, stoken: t.Optional[str], status_code=status.HTTP_409_CONFLICT, ) - report_items_changed(collection_object.uid, collection_object.stoken, data.items) + background_tasks.add_task(report_items_changed, collection_object.uid, collection_object.stoken, data.items) @item_router.get( @@ -564,20 +570,22 @@ def fetch_updates( def item_transaction( collection_uid: str, data: ItemBatchIn, + background_tasks: BackgroundTasks, stoken: t.Optional[str] = None, user: UserType = Depends(get_authenticated_user), ): - return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True) + return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True, background_tasks=background_tasks) @item_router.post("/item/batch/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) def item_batch( collection_uid: str, data: ItemBatchIn, + background_tasks: BackgroundTasks, stoken: t.Optional[str] = None, user: UserType = Depends(get_authenticated_user), ): - return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False) + return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False, background_tasks=background_tasks) # Chunks From 233aeab98b65118cb606954c242cee55557706ad Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 Jan 2021 09:35:36 +0200 Subject: [PATCH 486/511] Support login and invitations using email rather than just username. --- etebase_fastapi/routers/authentication.py | 4 ++-- etebase_fastapi/routers/invitation.py | 8 ++++---- etebase_fastapi/utils.py | 5 +++++ 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/etebase_fastapi/routers/authentication.py b/etebase_fastapi/routers/authentication.py index f8628b2..7ec6bc6 100644 --- a/etebase_fastapi/routers/authentication.py +++ b/etebase_fastapi/routers/authentication.py @@ -22,7 +22,7 @@ from django_etebase.utils import create_user, get_user_queryset, CallbackContext from myauth.models import UserType, get_typed_user_model from ..exceptions import AuthenticationFailed, transform_validation_error, HttpError from ..msgpack import MsgpackRoute -from ..utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode +from ..utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode, get_user_username_email_kwargs from ..dependencies import AuthData, get_auth_data, get_authenticated_user User = get_typed_user_model() @@ -114,7 +114,7 @@ class SignupIn(BaseModel): def get_login_user(request: Request, challenge: LoginChallengeIn) -> UserType: username = challenge.username - kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} + kwargs = get_user_username_email_kwargs(username) try: user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) user = user_queryset.get(**kwargs) diff --git a/etebase_fastapi/routers/invitation.py b/etebase_fastapi/routers/invitation.py index cbe570b..7e52978 100644 --- a/etebase_fastapi/routers/invitation.py +++ b/etebase_fastapi/routers/invitation.py @@ -12,6 +12,7 @@ from ..exceptions import HttpError, PermissionDenied from ..msgpack import MsgpackRoute from ..utils import ( get_object_or_404, + get_user_username_email_kwargs, Context, is_collection_admin, BaseModel, @@ -191,9 +192,8 @@ def outgoing_create( user: UserType = Depends(get_authenticated_user), ): collection = get_object_or_404(models.Collection.objects, uid=data.collection) - to_user = get_object_or_404( - get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), username=data.username - ) + kwargs = get_user_username_email_kwargs(data.username) + to_user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) context = Context(user, None) data.validate_db(context) @@ -238,7 +238,7 @@ def outgoing_fetch_user_profile( request: Request, user: UserType = Depends(get_authenticated_user), ): - kwargs = {User.USERNAME_FIELD: username.lower()} + kwargs = get_user_username_email_kwargs(username) user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) user_info = get_object_or_404(models.UserInfo.objects.all(), owner=user) return UserInfoOut.from_orm(user_info) diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py index 9f915e2..09c223e 100644 --- a/etebase_fastapi/utils.py +++ b/etebase_fastapi/utils.py @@ -69,6 +69,11 @@ def b64decode(data: str): return base64.urlsafe_b64decode(data) +def get_user_username_email_kwargs(username: str): + field_name = User.EMAIL_FIELD if "@" in username else User.USERNAME_FIELD + return {field_name + "__iexact": username.lower()} + + PERMISSIONS_READ = [Depends(x) for x in app_settings.API_PERMISSIONS_READ] PERMISSIONS_READWRITE = PERMISSIONS_READ + [Depends(x) for x in app_settings.API_PERMISSIONS_WRITE] From 0ac41e77e830fd5a64df6b46651cd1828d395fd7 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 Jan 2021 14:38:10 +0200 Subject: [PATCH 487/511] Fix DB errors when using MySQL/MariaDB. Fixes #69. --- .../migrations/0037_auto_20210127_1237.py | 18 ++++++++++++++++++ django_etebase/models.py | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 django_etebase/migrations/0037_auto_20210127_1237.py diff --git a/django_etebase/migrations/0037_auto_20210127_1237.py b/django_etebase/migrations/0037_auto_20210127_1237.py new file mode 100644 index 0000000..06c31b2 --- /dev/null +++ b/django_etebase/migrations/0037_auto_20210127_1237.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.1 on 2021-01-27 12:37 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0036_auto_20201214_1128'), + ] + + operations = [ + migrations.AlterField( + model_name='collectiontype', + name='uid', + field=models.BinaryField(db_index=True, editable=True, max_length=1024, unique=True), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index fa56a95..4640e6d 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -36,7 +36,7 @@ def stoken_annotation_builder(stoken_id_fields: t.List[str]): class CollectionType(models.Model): owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) - uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True) + uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True, max_length=1024) objects: models.manager.BaseManager["CollectionType"] From e4361d2364c3ab0c0b808cb4c546bf6d36e53694 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 27 Jan 2021 18:48:56 +0200 Subject: [PATCH 488/511] Patch old DB migration in order to fix mysql issues. Really fix #69. --- django_etebase/migrations/0032_auto_20201013_1409.py | 2 +- django_etebase/migrations/0037_auto_20210127_1237.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/django_etebase/migrations/0032_auto_20201013_1409.py b/django_etebase/migrations/0032_auto_20201013_1409.py index 2bb3cb0..c6e92a2 100644 --- a/django_etebase/migrations/0032_auto_20201013_1409.py +++ b/django_etebase/migrations/0032_auto_20201013_1409.py @@ -13,6 +13,6 @@ class Migration(migrations.Migration): migrations.AlterField( model_name="collectiontype", name="uid", - field=models.BinaryField(db_index=True, editable=True, unique=True), + field=models.BinaryField(db_index=True, editable=True, max_length=1024, unique=True), ), ] diff --git a/django_etebase/migrations/0037_auto_20210127_1237.py b/django_etebase/migrations/0037_auto_20210127_1237.py index 06c31b2..deaba1e 100644 --- a/django_etebase/migrations/0037_auto_20210127_1237.py +++ b/django_etebase/migrations/0037_auto_20210127_1237.py @@ -6,13 +6,13 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('django_etebase', '0036_auto_20201214_1128'), + ("django_etebase", "0036_auto_20201214_1128"), ] operations = [ migrations.AlterField( - model_name='collectiontype', - name='uid', + model_name="collectiontype", + name="uid", field=models.BinaryField(db_index=True, editable=True, max_length=1024, unique=True), ), ] From ef69954b6dd7b1e3e02a0fc148a1d1b9de2a4029 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 28 Jan 2021 17:44:17 +0200 Subject: [PATCH 489/511] requirements.txt: Add missing deps. --- requirements.in/base.txt | 2 ++ requirements.txt | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/requirements.in/base.txt b/requirements.in/base.txt index 44e2875..4b9b85b 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -4,3 +4,5 @@ pynacl fastapi typing_extensions uvicorn[standard] +aiofiles +aioredis diff --git a/requirements.txt b/requirements.txt index f59431b..213462f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,12 +4,16 @@ # # pip-compile --output-file=requirements.txt requirements.in/base.txt # +aiofiles==0.6.0 # via -r requirements.in/base.txt +aioredis==1.3.1 # via -r requirements.in/base.txt asgiref==3.3.1 # via django +async-timeout==3.0.1 # via aioredis cffi==1.14.4 # via pynacl click==7.1.2 # via uvicorn django==3.1.4 # via -r requirements.in/base.txt fastapi==0.63.0 # via -r requirements.in/base.txt h11==0.11.0 # via uvicorn +hiredis==1.1.0 # via aioredis httptools==0.1.1 # via uvicorn msgpack==1.0.2 # via -r requirements.in/base.txt pycparser==2.20 # via cffi From 848580604673f93fc54c9ea76b1aedd305b113af Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 28 Jan 2021 17:55:37 +0200 Subject: [PATCH 490/511] Easy config: add support for setting redis URI. --- etebase-server.ini.example | 1 + etebase_server/settings.py | 3 +++ 2 files changed, 4 insertions(+) diff --git a/etebase-server.ini.example b/etebase-server.ini.example index 2b4682a..13903b2 100644 --- a/etebase-server.ini.example +++ b/etebase-server.ini.example @@ -8,6 +8,7 @@ debug = false ;media_url = /user-media/ ;language_code = en-us ;time_zone = UTC +;redis_uri = redis://localhost:6379 [allowed_hosts] allowed_host1 = example.com diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 5d57ec0..80a6ee9 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -154,6 +154,9 @@ if any(os.path.isfile(x) for x in config_locations): TIME_ZONE = section.get("time_zone", TIME_ZONE) DEBUG = section.getboolean("debug", DEBUG) + if "redis_uri" in section: + ETEBASE_REDIS_URI = section.get("redis_uri") + if "allowed_hosts" in config: ALLOWED_HOSTS = [y for x, y in config.items("allowed_hosts")] From b33a3c882ee12e665f9ddfebbfa515315543f532 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 28 Jan 2021 18:10:12 +0200 Subject: [PATCH 491/511] Subscriptions: allow subscribing without setting an stoken. --- etebase_fastapi/routers/websocket.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/routers/websocket.py b/etebase_fastapi/routers/websocket.py index ad3331b..3fc535f 100644 --- a/etebase_fastapi/routers/websocket.py +++ b/etebase_fastapi/routers/websocket.py @@ -74,7 +74,7 @@ def get_websocket_user(websocket: WebSocket, ticket_model: t.Optional[TicketInne @websocket_router.websocket("/{ticket}/") async def websocket_endpoint( websocket: WebSocket, - stoken: t.Optional[str], + stoken: t.Optional[str] = None, user: t.Optional[UserType] = Depends(get_websocket_user), ticket_model: TicketInner = Depends(load_websocket_ticket), ): From 14b3cfca556ae117e0f28017a0b2516a19c1ec7d Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Mon, 1 Feb 2021 18:40:11 +0200 Subject: [PATCH 492/511] Handle stoken being the empty string. For whatever reason some users were getting this which was causing this code to fail. --- etebase_fastapi/routers/collection.py | 2 +- etebase_fastapi/stoken_handler.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py index 167179d..c8146f2 100644 --- a/etebase_fastapi/routers/collection.py +++ b/etebase_fastapi/routers/collection.py @@ -473,7 +473,7 @@ def item_bulk_common( with transaction.atomic(): # We need this for locking the collection object collection_object = queryset.select_for_update().get(uid=uid) - if stoken is not None and stoken != collection_object.stoken: + if stoken and stoken != collection_object.stoken: raise HttpError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) data.validate_db() diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py index 76d348a..b6f2999 100644 --- a/etebase_fastapi/stoken_handler.py +++ b/etebase_fastapi/stoken_handler.py @@ -12,7 +12,7 @@ StokenAnnotation = t.Any def get_stoken_obj(stoken: t.Optional[str]) -> t.Optional[Stoken]: - if stoken is not None: + if stoken: try: return Stoken.objects.get(uid=stoken) except Stoken.DoesNotExist: From 21e5382fc43b57edabf322e018eb746dc717bb37 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 17 Feb 2021 17:33:20 +0200 Subject: [PATCH 493/511] easyconfig: make it clear that media_root needs to be set. --- etebase-server.ini.example | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/etebase-server.ini.example b/etebase-server.ini.example index 13903b2..4f4b456 100644 --- a/etebase-server.ini.example +++ b/etebase-server.ini.example @@ -1,10 +1,12 @@ [global] secret_file = secret.txt debug = false +;Set the paths where data will be stored at +static_root = /path/to/static +media_root = /path/to/media + ;Advanced options, only uncomment if you know what you're doing: -;static_root = /path/to/static ;static_url = /static/ -;media_root = /path/to/media ;media_url = /user-media/ ;language_code = en-us ;time_zone = UTC From 58163d6678be29f80b930abe4bb6d2089945eb9c Mon Sep 17 00:00:00 2001 From: Zakkumaru Date: Sat, 6 Mar 2021 09:36:02 +0900 Subject: [PATCH 494/511] Duplicate to README.MD When the file was changed/renamed, it retained a duplicate, possibly outdated. --- example-configs/nginx-uwsgi/readme.md | 20 -------------------- 1 file changed, 20 deletions(-) delete mode 100644 example-configs/nginx-uwsgi/readme.md diff --git a/example-configs/nginx-uwsgi/readme.md b/example-configs/nginx-uwsgi/readme.md deleted file mode 100644 index dad98b6..0000000 --- a/example-configs/nginx-uwsgi/readme.md +++ /dev/null @@ -1,20 +0,0 @@ -# Running `etesync` under `nginx` and `uwsgi` - -This configuration assumes that etesync server has been installed in the home folder of a non privileged user -called `EtesyncUser` following the instructions in . Also that static -files have been collected at `/srv/http/etesync_server` by running the following commands: - - sudo mkdir -p /srv/http/etesync_server/static - sudo chown -R EtesyncUser /srv/http/etesync_server - sudo su EtesyncUser - cd /path/to/etesync - ln -s /srv/http/etesync_server/static static - ./manage.py collectstatic - -It is also assumed that `nginx` and `uwsgi` have been installed system wide by `root`, and that `nginx` is running as user/group `www-data`. - -In this setup, `uwsgi` running as a `systemd` service as `root` creates a unix socket with read-write access -to both `EtesyncUser` and `nginx`. It then drops its `root` privilege and runs `etesync` as `EtesyncUser`. - -`nginx` listens on the `https` port (or a non standard port `https` port if desired), delivers static pages directly -and for everything else, communicates with `etesync` over the unix socket. From 7c58540409d420d3c4fa971ffbf0f2ed895032b9 Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Fri, 16 Apr 2021 18:32:06 +0000 Subject: [PATCH 495/511] Create a testing docker image --- .dockerignore | 16 +++++++++++ README.md | 6 +++++ docker/build.sh | 16 +++++++++++ docker/test-server/Dockerfile | 38 +++++++++++++++++++++++++++ docker/test-server/entrypoint.sh | 6 +++++ docker/test-server/etebase-server.ini | 12 +++++++++ 6 files changed, 94 insertions(+) create mode 100644 .dockerignore create mode 100755 docker/build.sh create mode 100644 docker/test-server/Dockerfile create mode 100644 docker/test-server/entrypoint.sh create mode 100644 docker/test-server/etebase-server.ini diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..cd05353 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,16 @@ +/db.sqlite3* +Session.vim +/local_settings.py +.venv +/assets +/logs +/.coverage +/tmp +/media + +__pycache__ +.*.swp + +/.* + +/sandbox diff --git a/README.md b/README.md index 1d39872..7a7efac 100644 --- a/README.md +++ b/README.md @@ -138,6 +138,12 @@ Here are the update steps: 4. Run the migration tool to migrate all of your data. 5. Add your new EteSync 2.0 accounts to all of your devices. +# Testing + +Docker images named `etesync/test-server:` and `:latest` are available for testing etesync clients. +This docker image starts a server on port 3735 that supports user signup (without email confirmation), is in debug mode (thus supporting the reset endpoint), and stores its data locally. +It is in no way suitable for production usage, but is able to start up quickly and makes a good component of CI for etesync clients and users of those clients. + # License Etebase is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation. See the [LICENSE](./LICENSE) for more information. diff --git a/docker/build.sh b/docker/build.sh new file mode 100755 index 0000000..b8e97cc --- /dev/null +++ b/docker/build.sh @@ -0,0 +1,16 @@ +#! /bin/bash + +# Build the `test-server` image, which runs the server in a simple configuration +# designed to be used in tests, based on the current git revision. + +TAG="${1:-latest}" + +echo "Building working copy to etesync/test-server:${TAG}" + +ETESYNC_VERSION=$(git describe --tags) + +docker build \ + --build-arg ETESYNC_VERSION=${ETESYNC_VERSION} \ + -t etesync/test-server:${TAG} \ + -f docker/test-server/Dockerfile \ + . diff --git a/docker/test-server/Dockerfile b/docker/test-server/Dockerfile new file mode 100644 index 0000000..19ddb91 --- /dev/null +++ b/docker/test-server/Dockerfile @@ -0,0 +1,38 @@ +FROM python:3.9.0-alpine + +ARG ETESYNC_VERSION + +ENV PIP_DISABLE_PIP_VERSION_CHECK=1 +ENV PIP_NO_CACHE_DIR=1 + +# install packages and pip requirements first, in a single step, +COPY /requirements.txt /requirements.txt +RUN set -ex ;\ + apk add libpq postgresql-dev --virtual .build-deps coreutils gcc libc-dev libffi-dev make ;\ + pip install -U pip ;\ + pip install --no-cache-dir --progress-bar off -r /requirements.txt ;\ + apk del .build-deps make gcc coreutils ;\ + rm -rf /root/.cache + +COPY . /app + +RUN set -ex ;\ + mkdir -p /data/static /data/media ;\ + cd /app ;\ + mkdir -p /etc/etebase-server ;\ + cp docker/test-server/etebase-server.ini /etc/etebase-server ;\ + sed -e '/ETEBASE_CREATE_USER_FUNC/ s/^#*/#/' -i /app/etebase_server/settings.py ;\ + chmod +x docker/test-server/entrypoint.sh + +# this is a test image and should start up quickly, so it starts with the DB +# and static data already fully set up. +RUN set -ex ;\ + cd /app ;\ + python manage.py migrate ;\ + python manage.py collectstatic --noinput + +ENV ETESYNC_VERSION=${ETESYNC_VERSION} +VOLUME /data +EXPOSE 3735 + +ENTRYPOINT ["/app/docker/test-server/entrypoint.sh"] diff --git a/docker/test-server/entrypoint.sh b/docker/test-server/entrypoint.sh new file mode 100644 index 0000000..27b7fff --- /dev/null +++ b/docker/test-server/entrypoint.sh @@ -0,0 +1,6 @@ +#! /bin/sh + +echo "Running etesync test server ${ETESYNC_VERSION}" + +cd /app +uvicorn etebase_server.asgi:application --host 0.0.0.0 --port 3735 diff --git a/docker/test-server/etebase-server.ini b/docker/test-server/etebase-server.ini new file mode 100644 index 0000000..a9b65ff --- /dev/null +++ b/docker/test-server/etebase-server.ini @@ -0,0 +1,12 @@ +[global] +secret_file = secret.txt +debug = true +static_root = /data/static +media_root = /data/media + +[allowed_hosts] +allowed_host1 = * + +[database] +engine = django.db.backends.sqlite3 +name = /db.sqlite3 From 43d5af32d72d5f59de7f31698076becf2430ccaa Mon Sep 17 00:00:00 2001 From: "Dustin J. Mitchell" Date: Sat, 17 Apr 2021 15:22:09 +0000 Subject: [PATCH 496/511] Fix sendfile settings * set SENDFILE_ROOT to the filesystem path for media, not the URL component * use the correct import path to the sendfile backend --- etebase_server/settings.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etebase_server/settings.py b/etebase_server/settings.py index 80a6ee9..aa77461 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -166,8 +166,8 @@ if any(os.path.isfile(x) for x in config_locations): ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked" # Efficient file streaming (for large files) -SENDFILE_BACKEND = "django_etebase.sendfile.backends.simple" -SENDFILE_ROOT = MEDIA_URL +SENDFILE_BACKEND = "etebase_fastapi.sendfile.backends.simple" +SENDFILE_ROOT = MEDIA_ROOT # Make an `etebase_server_settings` module available to override settings. try: From d4de717cf7e64f21388cc2b1d6c12df277738275 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 28 Jul 2021 12:29:32 +0300 Subject: [PATCH 497/511] README: Add @jzacsh to supporters Thanks a lot for your support! --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 7a7efac..7484da4 100644 --- a/README.md +++ b/README.md @@ -160,6 +160,10 @@ Please consider registering an account even if you self-host in order to support Become a financial contributor and help us sustain our community! +## Supporters ($20 / month) + +[![jzacsh](https://github.com/jzacsh.png?size=80)](https://github.com/jzacsh) + ## Contributors ($10 / month) [![ilovept](https://github.com/ilovept.png?size=40)](https://github.com/ilovept) From d11504093c0dff22e5371fada9037e6436abf628 Mon Sep 17 00:00:00 2001 From: Mohammed Anas Date: Thu, 29 Jul 2021 17:10:56 +0000 Subject: [PATCH 498/511] Make it clear in README that backing up secret.txt is ok --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7484da4..9e825fb 100644 --- a/README.md +++ b/README.md @@ -109,9 +109,10 @@ The default configuration creates a file “`secret.txt`” in the project’s base directory, which is used as the value of the Django `SECRET_KEY` setting. You can revoke this key by deleting the `secret.txt` file and the next time the app is run, a new one will be generated. Make sure you keep -the `secret.txt` file secret (don’t accidentally commit it to version -control, exclude it from your backups, etc.). If you want to change to a -more secure system for storing secrets, edit `etesync_server/settings.py` +the `secret.txt` file secret (e.g. don’t accidentally commit it to version +control). However, backing it up is okay, and it makes it easier to restore +the database to a new EteSync server, but it's not essential. If you want to +change to a more secure system for storing secrets, edit `etesync_server/settings.py` and implement your own method for setting `SECRET_KEY` (remove the line where it uses the `get_secret_from_file` function). Read the Django docs for more information about the `SECRET_KEY` and its uses. From 453869d71d04b2bc454126e60515aa09a7bcb8b9 Mon Sep 17 00:00:00 2001 From: James Date: Sun, 19 Sep 2021 14:21:33 -0400 Subject: [PATCH 499/511] Remove port from host_from_request check This strips the port from the client requests, comparing only the hostnames or IP addresses, and should alleviate nonstandard port and initial testing issues. Tested on Linux, Mac, and Android clients. Closes #105. Code is from this comment: https://github.com/etesync/server/issues/66#issuecomment-731215345 --- etebase_fastapi/routers/authentication.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/routers/authentication.py b/etebase_fastapi/routers/authentication.py index 7ec6bc6..fd21d21 100644 --- a/etebase_fastapi/routers/authentication.py +++ b/etebase_fastapi/routers/authentication.py @@ -161,7 +161,7 @@ def validate_login_request( raise HttpError("challenge_expired", "Login challenge has expired") elif challenge_data["userId"] != user.id: raise HttpError("wrong_user", "This challenge is for the wrong user") - elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request: + elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request.split(":", 1)[0]: raise HttpError( "wrong_host", f'Found wrong host name. Got: "{validated_data.host}" expected: "{host_from_request}"' ) From 4c4fa3d7264215d5168d1995e1f2b484132ec417 Mon Sep 17 00:00:00 2001 From: Simon Vandevelde Date: Thu, 11 Nov 2021 21:27:44 +0100 Subject: [PATCH 500/511] Update README.md with automatic user signup --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index 9e825fb..06b9efc 100644 --- a/README.md +++ b/README.md @@ -145,6 +145,14 @@ Docker images named `etesync/test-server:` and `:latest` are available This docker image starts a server on port 3735 that supports user signup (without email confirmation), is in debug mode (thus supporting the reset endpoint), and stores its data locally. It is in no way suitable for production usage, but is able to start up quickly and makes a good component of CI for etesync clients and users of those clients. +# User signup + +Instead of having to create Django users manually when signup up Etebase users, it is also possible to allow automatic signup. +For example, this makes sense when putting an Etebase server in production. +However, this does come with the added risk that everybody with access to your server will be able to sign up. + +In order to set it up, comment out the line `ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked"` in `server/settings.py` and restart your Etebase server. + # License Etebase is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation. See the [LICENSE](./LICENSE) for more information. From 056d6853a0a9f1c7918e0d7d8a4ae9b8fea393ed Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 17 Mar 2022 11:47:41 +0200 Subject: [PATCH 501/511] Deps: update django dep. This is in response to reports in #123. There are no security issues affecting Etebase, but people still misunderstood the reports so updating to make sure that it's clear. More info: https://github.com/etesync/server/issues/123\#issuecomment-1070644715 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 213462f..3fd8230 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,7 +10,7 @@ asgiref==3.3.1 # via django async-timeout==3.0.1 # via aioredis cffi==1.14.4 # via pynacl click==7.1.2 # via uvicorn -django==3.1.4 # via -r requirements.in/base.txt +django==3.2.12 # via -r requirements.in/base.txt fastapi==0.63.0 # via -r requirements.in/base.txt h11==0.11.0 # via uvicorn hiredis==1.1.0 # via aioredis From f14d74510b9e277ef9f4682a12b542c41855796a Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 17 Mar 2022 20:25:37 +0200 Subject: [PATCH 502/511] Update changelog. --- ChangeLog.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index a74a8af..df93b74 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,14 @@ # Changelog +## Version 0.8.0 +* Update django dep. +* Fix issue with comparing ports in hostname verification with self-hosted servers. +* Fix sendfile settings to be more correct. +* Improve easy config (make it clear media_root needs to be set) +* Handle stoken being the empty string +* Fix mysql/mariadb support +* Switch to FastAPI for the server component + ## Version 0.7.0 * Chunks: improve the chunk download endpoint to use sendfile extensions * Chunks: support not passing chunk content if exists From ee8349d419f44e50b6c183b3f3022e4a3c86e04b Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Thu, 17 Mar 2022 20:27:27 +0200 Subject: [PATCH 503/511] Update django version in requirements-dev.txt too --- requirements-dev.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index 15a8b60..2fd7f88 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -10,7 +10,7 @@ black==20.8b1 # via -r requirements.in/development.txt click==7.1.2 # via black, pip-tools coverage==5.3.1 # via -r requirements.in/development.txt django-stubs==1.7.0 # via -r requirements.in/development.txt -django==3.1.4 # via django-stubs +django==3.2.12 # via django-stubs mypy-extensions==0.4.3 # via black, mypy mypy==0.790 # via django-stubs pathspec==0.8.1 # via black From ce70045dac251ede9e308d1cb5a63d80bd2b87ea Mon Sep 17 00:00:00 2001 From: "Victor R. Santos" Date: Sun, 20 Mar 2022 12:21:09 +0000 Subject: [PATCH 504/511] Fix Error `404 Not Found` for Static Files (#124) After changing for FastAPI the server ignores STATIC_ROOT and STATIC_URL --- etebase_fastapi/main.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py index d63c01d..3e0c1e9 100644 --- a/etebase_fastapi/main.py +++ b/etebase_fastapi/main.py @@ -4,6 +4,7 @@ from django.conf import settings from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.trustedhost import TrustedHostMiddleware +from fastapi.staticfiles import StaticFiles from django_etebase import app_settings @@ -74,4 +75,6 @@ def create_application(prefix="", middlewares=[]): async def custom_exception_handler(request: Request, exc: CustomHttpException): return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) + app.mount(settings.STATIC_URL, StaticFiles(directory=settings.STATIC_ROOT), name="static") + return app From d1d58f15c71b949bc989e544c24f0bce041e1f1c Mon Sep 17 00:00:00 2001 From: "Victor R. Santos" Date: Sat, 19 Mar 2022 16:20:03 -0300 Subject: [PATCH 505/511] Update dependencies while keeping Django below 4.0 --- requirements-dev.txt | 70 ++++++++++++++++++++--------- requirements.in/base.txt | 2 +- requirements.in/development.txt | 1 + requirements.txt | 80 ++++++++++++++++++++++----------- 4 files changed, 105 insertions(+), 48 deletions(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index 2fd7f88..4f99acf 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,28 +1,58 @@ # -# This file is autogenerated by pip-compile +# This file is autogenerated by pip-compile with python 3.9 # To update, run: # # pip-compile --output-file=requirements-dev.txt requirements.in/development.txt # -appdirs==1.4.4 # via black -asgiref==3.3.1 # via django -black==20.8b1 # via -r requirements.in/development.txt -click==7.1.2 # via black, pip-tools -coverage==5.3.1 # via -r requirements.in/development.txt -django-stubs==1.7.0 # via -r requirements.in/development.txt -django==3.2.12 # via django-stubs -mypy-extensions==0.4.3 # via black, mypy -mypy==0.790 # via django-stubs -pathspec==0.8.1 # via black -pip-tools==5.4.0 # via -r requirements.in/development.txt -pytz==2020.5 # via django -pywatchman==1.4.1 # via -r requirements.in/development.txt -regex==2020.11.13 # via black -six==1.15.0 # via pip-tools -sqlparse==0.4.1 # via django -toml==0.10.2 # via black -typed-ast==1.4.1 # via black, mypy -typing-extensions==3.7.4.3 # via black, django-stubs, mypy +appdirs==1.4.4 + # via black +asgiref==3.5.0 + # via django +black==20.8b1 + # via -r requirements.in/development.txt +click==7.1.2 + # via + # black + # pip-tools +coverage==5.3.1 + # via -r requirements.in/development.txt +django==3.2.12 + # via + # -r requirements.in/development.txt + # django-stubs +django-stubs==1.7.0 + # via -r requirements.in/development.txt +mypy==0.790 + # via django-stubs +mypy-extensions==0.4.3 + # via + # black + # mypy +pathspec==0.8.1 + # via black +pip-tools==5.4.0 + # via -r requirements.in/development.txt +pytz==2020.5 + # via django +pywatchman==1.4.1 + # via -r requirements.in/development.txt +regex==2020.11.13 + # via black +six==1.15.0 + # via pip-tools +sqlparse==0.4.1 + # via django +toml==0.10.2 + # via black +typed-ast==1.4.1 + # via + # black + # mypy +typing-extensions==3.7.4.3 + # via + # black + # django-stubs + # mypy # The following packages are considered to be unsafe in a requirements file: # pip diff --git a/requirements.in/base.txt b/requirements.in/base.txt index 4b9b85b..e253c5e 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -1,4 +1,4 @@ -django +django<4.0 msgpack pynacl fastapi diff --git a/requirements.in/development.txt b/requirements.in/development.txt index fb281d3..bf565de 100644 --- a/requirements.in/development.txt +++ b/requirements.in/development.txt @@ -3,3 +3,4 @@ pip-tools pywatchman black django-stubs +django<4.0 diff --git a/requirements.txt b/requirements.txt index 3fd8230..58486d6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,32 +1,58 @@ # -# This file is autogenerated by pip-compile +# This file is autogenerated by pip-compile with python 3.9 # To update, run: # # pip-compile --output-file=requirements.txt requirements.in/base.txt # -aiofiles==0.6.0 # via -r requirements.in/base.txt -aioredis==1.3.1 # via -r requirements.in/base.txt -asgiref==3.3.1 # via django -async-timeout==3.0.1 # via aioredis -cffi==1.14.4 # via pynacl -click==7.1.2 # via uvicorn -django==3.2.12 # via -r requirements.in/base.txt -fastapi==0.63.0 # via -r requirements.in/base.txt -h11==0.11.0 # via uvicorn -hiredis==1.1.0 # via aioredis -httptools==0.1.1 # via uvicorn -msgpack==1.0.2 # via -r requirements.in/base.txt -pycparser==2.20 # via cffi -pydantic==1.7.3 # via fastapi -pynacl==1.4.0 # via -r requirements.in/base.txt -python-dotenv==0.15.0 # via uvicorn -pytz==2020.4 # via django -pyyaml==5.3.1 # via uvicorn -six==1.15.0 # via pynacl -sqlparse==0.4.1 # via django -starlette==0.13.6 # via fastapi -typing-extensions==3.7.4.3 # via -r requirements.in/base.txt -uvicorn[standard]==0.13.2 # via -r requirements.in/base.txt -uvloop==0.14.0 # via uvicorn -watchgod==0.6 # via uvicorn -websockets==8.1 # via uvicorn +aiofiles==0.6.0 + # via -r requirements.in/base.txt +aioredis==1.3.1 + # via -r requirements.in/base.txt +asgiref==3.5.0 + # via django +async-timeout==3.0.1 + # via aioredis +cffi==1.14.4 + # via pynacl +click==7.1.2 + # via uvicorn +django==3.2.12 + # via -r requirements.in/base.txt +fastapi==0.63.0 + # via -r requirements.in/base.txt +h11==0.11.0 + # via uvicorn +hiredis==1.1.0 + # via aioredis +httptools==0.1.1 + # via uvicorn +msgpack==1.0.2 + # via -r requirements.in/base.txt +pycparser==2.20 + # via cffi +pydantic==1.7.3 + # via fastapi +pynacl==1.4.0 + # via -r requirements.in/base.txt +python-dotenv==0.15.0 + # via uvicorn +pytz==2020.4 + # via django +pyyaml==5.3.1 + # via uvicorn +six==1.15.0 + # via pynacl +sqlparse==0.4.1 + # via django +starlette==0.13.6 + # via fastapi +typing-extensions==3.7.4.3 + # via -r requirements.in/base.txt +uvicorn[standard]==0.13.2 + # via -r requirements.in/base.txt +uvloop==0.14.0 + # via uvicorn +watchgod==0.6 + # via uvicorn +websockets==8.1 + # via uvicorn From 7bb1bf9d2233152dbc97fcbd860ee3b28600dbfa Mon Sep 17 00:00:00 2001 From: "Victor R. Santos" Date: Sat, 19 Mar 2022 16:21:22 -0300 Subject: [PATCH 506/511] Fix Django 3.2 warnings models.W042 --- etebase_server/settings.py | 1 + 1 file changed, 1 insertion(+) diff --git a/etebase_server/settings.py b/etebase_server/settings.py index aa77461..42ded0b 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -43,6 +43,7 @@ DATABASES = { } } +DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' # Application definition From ed2e68d4d52e0ece192750ba43e62c2efde0f648 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Sun, 20 Mar 2022 17:48:23 +0200 Subject: [PATCH 507/511] Update changelog --- ChangeLog.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index df93b74..f567869 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,10 @@ # Changelog +## Version 0.8.1 +* Fix Error `404 Not Found` for Static Files +* Fix Django 3.2 warnings +* Update dependencies while (keep Django 3.2 LTS) + ## Version 0.8.0 * Update django dep. * Fix issue with comparing ports in hostname verification with self-hosted servers. From e0010f21f6e894380c07ecca2419cea24fae3718 Mon Sep 17 00:00:00 2001 From: "Victor R. Santos" Date: Wed, 23 Mar 2022 09:36:21 +0000 Subject: [PATCH 508/511] Update dependecies generated by pip-compile. (#126) This one is to fix my own PR #125, the requirement files contained the following sentence: "To update, run: pip-compile --output-file=requirements.txt requirements.in/base.txt" But that was misleading, after reading pip-tools documentation I found that "If pip-compile finds an existing requirements.txt file that fulfils the dependencies then no changes will be made, even if updates are available." That was my mistake, generated the files again and made new builds using python 3.10 that worked as expected. Once again sorry for the mistake! --- requirements-dev.txt | 49 +++++++++++++++++++------------- requirements.txt | 67 +++++++++++++++++++++++++------------------- 2 files changed, 67 insertions(+), 49 deletions(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index 4f99acf..2be0ac0 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,58 +1,67 @@ # -# This file is autogenerated by pip-compile with python 3.9 +# This file is autogenerated by pip-compile with python 3.10 # To update, run: # # pip-compile --output-file=requirements-dev.txt requirements.in/development.txt # -appdirs==1.4.4 - # via black asgiref==3.5.0 # via django -black==20.8b1 +black==22.1.0 # via -r requirements.in/development.txt -click==7.1.2 +click==8.0.4 # via # black # pip-tools -coverage==5.3.1 +coverage==6.3.2 # via -r requirements.in/development.txt django==3.2.12 # via # -r requirements.in/development.txt # django-stubs -django-stubs==1.7.0 + # django-stubs-ext +django-stubs==1.9.0 # via -r requirements.in/development.txt -mypy==0.790 +django-stubs-ext==0.3.1 + # via django-stubs +mypy==0.941 # via django-stubs mypy-extensions==0.4.3 # via # black # mypy -pathspec==0.8.1 +pathspec==0.9.0 # via black -pip-tools==5.4.0 +pep517==0.12.0 + # via pip-tools +pip-tools==6.5.1 # via -r requirements.in/development.txt -pytz==2020.5 +platformdirs==2.5.1 + # via black +pytz==2022.1 # via django pywatchman==1.4.1 # via -r requirements.in/development.txt -regex==2020.11.13 - # via black -six==1.15.0 - # via pip-tools -sqlparse==0.4.1 +sqlparse==0.4.2 # via django toml==0.10.2 - # via black -typed-ast==1.4.1 + # via django-stubs +tomli==2.0.1 # via # black # mypy -typing-extensions==3.7.4.3 + # pep517 +types-pytz==2021.3.6 + # via django-stubs +types-pyyaml==6.0.5 + # via django-stubs +typing-extensions==4.1.1 # via - # black # django-stubs + # django-stubs-ext # mypy +wheel==0.37.1 + # via pip-tools # The following packages are considered to be unsafe in a requirements file: # pip +# setuptools diff --git a/requirements.txt b/requirements.txt index 58486d6..45c96d4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,58 +1,67 @@ # -# This file is autogenerated by pip-compile with python 3.9 +# This file is autogenerated by pip-compile with python 3.10 # To update, run: # # pip-compile --output-file=requirements.txt requirements.in/base.txt # -aiofiles==0.6.0 +aiofiles==0.8.0 # via -r requirements.in/base.txt -aioredis==1.3.1 +aioredis==2.0.1 # via -r requirements.in/base.txt +anyio==3.5.0 + # via + # starlette + # watchgod asgiref==3.5.0 - # via django -async-timeout==3.0.1 + # via + # django + # uvicorn +async-timeout==4.0.2 # via aioredis -cffi==1.14.4 +cffi==1.15.0 # via pynacl -click==7.1.2 +click==8.0.4 # via uvicorn django==3.2.12 # via -r requirements.in/base.txt -fastapi==0.63.0 +fastapi==0.75.0 # via -r requirements.in/base.txt -h11==0.11.0 +h11==0.13.0 # via uvicorn -hiredis==1.1.0 - # via aioredis -httptools==0.1.1 +httptools==0.4.0 # via uvicorn -msgpack==1.0.2 +idna==3.3 + # via anyio +msgpack==1.0.3 # via -r requirements.in/base.txt -pycparser==2.20 +pycparser==2.21 # via cffi -pydantic==1.7.3 +pydantic==1.9.0 # via fastapi -pynacl==1.4.0 +pynacl==1.5.0 # via -r requirements.in/base.txt -python-dotenv==0.15.0 +python-dotenv==0.19.2 # via uvicorn -pytz==2020.4 +pytz==2022.1 # via django -pyyaml==5.3.1 +pyyaml==6.0 # via uvicorn -six==1.15.0 - # via pynacl -sqlparse==0.4.1 +sniffio==1.2.0 + # via anyio +sqlparse==0.4.2 # via django -starlette==0.13.6 +starlette==0.17.1 # via fastapi -typing-extensions==3.7.4.3 - # via -r requirements.in/base.txt -uvicorn[standard]==0.13.2 +typing-extensions==4.1.1 + # via + # -r requirements.in/base.txt + # aioredis + # pydantic +uvicorn[standard]==0.17.6 # via -r requirements.in/base.txt -uvloop==0.14.0 +uvloop==0.16.0 # via uvicorn -watchgod==0.6 +watchgod==0.8.1 # via uvicorn -websockets==8.1 +websockets==10.2 # via uvicorn From 247c5ea6807b86b5e0ad9b1e048de84f7ea1c2bb Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Wed, 23 Mar 2022 12:59:41 +0200 Subject: [PATCH 509/511] Update changelog. --- ChangeLog.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index f567869..2702802 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,8 @@ # Changelog +## Version 0.8.2 +- Update dependencies again + ## Version 0.8.1 * Fix Error `404 Not Found` for Static Files * Fix Django 3.2 warnings From f62d4ebdfc9ab8dc069015060b70b267143e78ba Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 25 Mar 2022 15:17:34 +0300 Subject: [PATCH 510/511] Msgpack handling: fix compatibilty with newer fastapi. Fixes #112. Supersedes #127. --- etebase_fastapi/msgpack.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py index a671e79..3871d15 100644 --- a/etebase_fastapi/msgpack.py +++ b/etebase_fastapi/msgpack.py @@ -12,7 +12,7 @@ from .db_hack import django_db_cleanup_decorator class MsgpackRequest(Request): media_type = "application/msgpack" - async def json(self) -> bytes: + async def body(self) -> bytes: if not hasattr(self, "_json"): body = await super().body() self._json = msgpack_decode(body) From 006c5fc2425f5aa060d2911c175dae6b6b1a19f3 Mon Sep 17 00:00:00 2001 From: Tom Hacohen Date: Fri, 25 Mar 2022 15:18:55 +0300 Subject: [PATCH 511/511] Update changelog. --- ChangeLog.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index 2702802..18bf06d 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,8 @@ # Changelog +## Version 0.8.3 +- Fix compatibility with latest fastapi + ## Version 0.8.2 - Update dependencies again