Merge branch 'update-dependencies-jt'

This will be release 1.4.

Addresses issues #4, #16, #27, #29, #30, and #40.

Author: jthomale

README.md

@@ -1046,51 +1046,61 @@ file is in `.gitignore` for that reason. There is a
 Early in development we implemented a series of tests using the built-in Django
 test runner to do some simple sanity-checking to make sure the Django ORM
 models for Sierra match the structures actually in the production database.
-You can still run these tests after you first install the catalog-api to ensure
-the models match your Sierra setup. Systems may differ from institution to
+We have since converted these to run via pytest: see
+`django/sierra/base/tests/test_database.py`.
+
+When you run the full test suite, as [described below](#unit-tests), these run
+against the test Sierra database—which is useful. But, there are times
+that you'll want to run these tests against your live database to make sure the
+models are accurate. For instance, systems may differ from institution to
 institution based on what products you have, so you may end up needing to fork
 this project and update the models so they work with your own setup. It may
 also be worth running these tests after Sierra upgrades so that you can make
 sure there were no changes made to the database that break the models.
 
-If using Docker, run:
+If using Docker, run _only_ the database tests using the following:
+
+    ./docker-compose.sh run --rm live-db-test
 
-    ./docker-compose.sh run manage-dev test base
+If not using Docker, you can use the below command, instead. If applicable,
+replace the value of the `--ds` option with whatever your DEV settings file is.
 
-If not using Docker, you can run:
+    cd <project_root>
+    pytest --ds=sierra.settings.dev django/sierra/base/tests/test_database.py
 
-    cd <project_root>/django/sierra
-    manage.py test base
+Note: Some of these tests may fail simply because the models are generally more
+restrictive than the live Sierra database. We are forcing ForeignKey-type
+relationships on a lot of fields that don't seem to have actual
+database-enforced keys in Sierra. E.g., from what I can gather, `id` fields are
+usually proper keys, while `code` fields may not be&mdash;but `code` fields are
+frequently used in a foreign-key-like capacity. I think this leads to a lot of
+the invalid codes you have in Sierra, where you have a code in a record that
+_should_ point to some entry in an administrative table (like a location), but
+it doesn't because the administrative table entry was deleted and the record
+was never updated. And there are other cases, as well. E.g., a code might use
+the string `none` instead of a null value, but there is no corresponding entry
+for `none` in the related table. Bib locations use the string `multi` to
+indicate that they have multiple locations, but there is no corresponding
+`multi` record in the location table. Etc.
 
-Note: If any `*_maps_to_database` tests fail, it indicates that there are
-fields on the model that aren't present in the database. These are more
-serious (but often easier to fix) than `*_sanity_check` tests, which test
-to ensure that relationship fields work properly. In either case, you can
-check the models against the SierraDNA documentation and your own database
-to see where the problems lie and decide if they're worth trying to fix in
-the models. If tests fail on models that are central, like `RecordMetadata`
-or `BibRecord`, then it's a problem. If tests fail on models that are more
-peripheral, like `LocationChange`, then finding and fixing those problems
-may be less of a priority, especially if you never intend to use those
-models in your API.
+Ultimately, even though these `code` relationships aren't database-enforced
+keys, we do still want the ORM to handle the relationships for us in the
+general case where you _can_ match a code with the entry that describes it.
+Otherwise we'd have to do the matching manually, which would somewhat reduce
+the utility of the ORM.
 
-Presumably because the Sierra data that customers have access to is
-implemented in views instead of tables, proper data integrity is lacking in
-a few cases. (E.g., the views sometimes don't implement proper primary key
-/ foreign key relationships.) This can cause `*_sanity_check` tests to fail
-in practice on live data when they should work in theory.
 
 ### <a name="unit-tests"></a>Running Unit(ish) Tests
 
 More recently we've been working on adding unit (and integration) tests that
-run with py.test. We recommend running these tests via Docker, but it *is*
+run with pytest. We recommend running these tests via Docker, but it *is*
 possible to run them outside of Docker if you're motivated enough.
 
 If you followed the [Docker setup](#installation-docker),
 
      ./docker-compose.sh run --rm test
 
-will run all available py.test tests.
+will run all available pytest tests.
 
 If you didn't follow the Docker setup, then you should still be able to create
 a comparable test environment:
@@ -1108,7 +1118,7 @@ relevant connection details.
 
 Spin up all of the needed test databases, and then run
 
-    py.test
+    pytest
 
 
 ### <a name="testing-exports"></a>Testing Sierra Exports Manually

django/sierra/api/__init__.py

@@ -1,164 +0,0 @@
-import csv
-import re
-
-from .models import APIUser
-from django.contrib.auth.models import User
-
-
-class APIUserManagerError(Exception):
-    pass
-
-
-class UserExists(APIUserManagerError):
-    pass
-
-
-class UserDoesNotExist(APIUserManagerError):
-    pass
-
-
-def get_api_user(username):
-    api_user = None
-    try:
-        api_user = APIUser.objects.get(user__username=username)
-    except APIUser.DoesNotExist:
-        raise APIUserManagerError('APIUser {} does not exist.'
-                                  ''.format(username))
-        
-    return api_user
-
-
-def create_api_user(username, secret, permissions=None, default=False, 
-                    email='', django_password=None, first_name='',
-                    last_name=''):
-    user = None
-    api_user = None
-
-    try:
-        api_user = get_api_user(username)
-    except APIUserManagerError:
-        pass
-    else:
-        raise UserExists('APIUser {} already exists.'.format(username))
-
-    try:
-        user = User.objects.get(username=username)
-    except User.DoesNotExist:
-        if not django_password:
-            raise UserDoesNotExist('Django User {} does not exist and '
-                                   'must be created. You must supply a '
-                                   'django_password argument.'
-                                   ''.format(username))
-        user = User.objects.create_user(username, email, django_password)
-        user.first_name = first_name
-        user.last_name = last_name
-        user.save()
-
-    api_user = APIUser(user=user)
-    api_user.set_secret(secret)
-    api_user.set_permissions(permissions=permissions, default=default)
-    return api_user
-
-
-def update_api_user(username, secret=None, permissions=None, default=False,
-                    email='', django_password=None, first_name='',
-                    last_name=''):
-    try:
-        api_user = get_api_user(username)
-    except APIUserManagerError:
-        raise UserDoesNotExist('API User {} does not exist.'.format(username))
-
-    if (secret):
-        api_user.set_secret(secret)
-    if (permissions):
-        api_user.set_permissions(permissions=permissions, default=default)
-        
-    django_user_changed = False
-    if (email and email != api_user.user.email):
-        api_user.user.email = email
-        django_user_changed = True
-    if (first_name and first_name != api_user.user.first_name):
-        api_user.user.first_name = first_name
-        django_user_changed = True
-    if (last_name and last_name != api_user.user.last_name):
-        api_user.user.last_name = last_name
-        django_user_changed = True
-    if (django_password and not api_user.user.check_password(django_password)):
-        api_user.user.set_password(django_password)
-        django_user_changed = True
-
-    if django_user_changed:
-        api_user.user.save()
-    return api_user
-
-
-def set_api_user_secret(username, secret):
-    api_user = get_api_user(username)
-    api_user.set_secret(secret)
-    return api_user
-
-
-def set_api_user_permissions(username, permissions, default=False):
-    api_user = get_api_user(username)
-    api_user.set_permissions(permissions=permissions, default=default)
-    return api_user
-
-
-def batch_create_update_api_users(filepath, default=None):
-    """
-    This will open a csv file provided by filepath and batch create API
-    users based on its contents. Column names should be provided that
-    match the args to create_api_user (except default). If the API user
-    already exists, then this will attempt to update the secret and the
-    permissions for that user.
-    """
-    data = []
-    with open(filepath, 'r') as csvfile:
-        permreader = csv.DictReader(csvfile)
-        for row in permreader:
-            row_data = {}
-            try:
-                row_data['username'] = row['username']
-            except KeyError:
-                raise APIUserManagerError('Your csv file must include a '
-                                          '"username" column.')
-            row_data['secret'] = row.get('secret', None)
-            row_data['django_password'] = row.get('django_password', None)
-            row_data['first_name'] = row.get('first_name', None)
-            row_data['last_name'] = row.get('last_name', None)
-            row_data['email'] = row.get('email', None)
-
-            permissions = {}
-
-            for key, val in row.iteritems():
-                if re.match(r'^permission_', key):
-                    if val in ('True', 'true', 'TRUE', 'T', 't', '1'):
-                        val = True
-                    else:
-                        val = False
-                    permissions[re.sub(r'^permission_', '', key)] = val
-
-            if not permissions and default is not None:
-                permissions = default
-
-            row_data['permissions'] = permissions
-            data.append(row_data)
-
-    for row in data:
-        try:
-            create_api_user(row['username'], row['secret'], 
-                            permissions=row['permissions'],
-                            email=row['email'],
-                            django_password=row['django_password'],
-                            first_name=row['first_name'],
-                            last_name=row['last_name'], default=default)
-        except UserExists:
-            update_api_user(row['username'], row['secret'],
-                            permissions=row['permissions'],
-                            email=row['email'],
-                            django_password=row['django_password'],
-                            first_name=row['first_name'],
-                            last_name=row['last_name'], default=default)
-        except UserDoesNotExist:
-            print ('User {} does not exist and no django_password was '
-                    'provided. Skipping.'.format(row['username']))

django/sierra/api/exceptions.py

@@ -1,31 +1,34 @@
-'''
+"""
 Implements REST API exceptions.
-'''
+"""
 from rest_framework import exceptions
 from rest_framework import views
 
-def sierra_exception_handler(exc):
-    '''
+
+def sierra_exception_handler(exc, context):
+    """
     Custom exception handler. Defines what gets returned in the
     response when clients encounter errors.
-    '''
-    response = views.exception_handler(exc)
+    """
+    response = views.exception_handler(exc, context)
     if response is not None:
         response.data['status'] = response.status_code
-
     return response
 
 
 class BadQuery(exceptions.APIException):
     status_code = 400
     default_detail = ('One of the query parameters was not correctly '
-                        'formatted.')
+                      'formatted.')
+
 
 class BadUpdate(exceptions.APIException):
     status_code = 400
     default_detail = ('The requested resource could not be updated because '
                       'the content received was invalid.')
+
+
 class ReadOnlyEdit(exceptions.APIException):
     status_code = 400
     default_detail = ('The requested resource could not be updated because '
-                      'the request attempted to update read-only content.')
+                      'the request attempted to update read-only content.')