Merge branch 'improve_documentation' into 'master'

fix bad markdown formatting and improve documentation

See merge request lap/alma-user-load!48

Author: steve-sullivan

README.md

@@ -1,57 +1,93 @@
 
 # Alma-User-Load
 
-Application for fetching patron records for both SIS (aka students) and UCPath (aka employees) and formatting those records into XML to be loaded by Ex Libris into Alma using their sync process. 
+Application for fetching patron records for both studens and employees and formatting those records into XML to be loaded by Ex Libris into Alma using their sync process. 
 
-Ex Libris pulls the XML files from our SFTP server.
-UCPath: upload.lib.berkeley.edu/alma/patron_employees
-SIS: upload.lib.berkeley.edu/alma/patron_students
+Additional documentation can be found on B-Drive: [link](https://drive.google.com/drive/folders/1pOEKi2d5SQ4VZpwjQzUwuArwBbUJzvcF)
 
-Additional documentation can be found on B-Drive: [link](https://drive.google.com/drive/folders/1qO_7oD4tzDO9H4UMHrms7-ONzHdQ4gU2)
+### Important Note
+Because of differences in the two APIs (SIS and UCPath), these do not operate exactly the same. UCPath's API is driven by a "change log". When it runs, it looks back 7 days from the current date for any recods that have been added or updated and then processes those. SIS does not have a change log. Instead we pull the entire collection of users for the current semester everytime we run SIS.
 
-___
+If the SIS Alma-User-Load fails to run on it's scheduled day, it's not considered detrimental - the next scheduled run will pick up any new or updated records the previous run would have picked up.
 
-#### Info
-Alma-User_load is deployed via the Docker Swarm.
-It is kept in the `lap/alma-user-load` repo.
+If UCPath **fails to run on it's scheduled day**, a manual run is necessary. The `startdate` (see Commandline Options below) needs to be the date 7 days prior to the scheduled date that was missed.  (e.g., if UCPath was scheduled to run on April 10, 2025 and failed and you wanted to run it on April 11th, you'd need to pass -startdate 2025-04-03 as a commandline arguement).
 
-#### Schedule
-SIS : Monday and Wednesday Mornings 1am
-UCPath : Thursday Mornings 1am
+You can perform that locally and manually copy the zipped file to the patron_employees folder (see XML and Zip File Output below)
 
-The schedules are defined in the `ops/docker-swarm` repo:
-`files/production/swarm/stacks/alma_user_loader.yml`
+## Info
 
-#### XML and Zip File Output
-Files are saved to:
-SIS: ```upload.lib.berkeley.edu/alma/patron_students```
+### Dependencies
+
+Alma-User_load is deployed via the Docker Swarm and is kept in the `lap/alma-user-load` repo.  
+Data is harvested from several sources:  
+
+UCPath: https://developers.api.berkeley.edu/api/8  
+SIS: https://developers.api.berkeley.edu/api/6  
+Additional data is pulled from CalNet via LDAP
+
+### Schedule
+
+* SIS : Monday and Wednesday Mornings 1am
+* UCPath : Thursday Mornings 1am
+
+The schedules are defined in the `ops/docker-swarm`  
+repo: `files/production/swarm/stacks/alma_user_loader.yml`
+
+### XML and Zip File Output
+
+Files are saved to:  
+SIS: ```upload.lib.berkeley.edu/alma/patron_students```  
 UCPath: ```upload.lib.berkeley.edu/alma/patron_employees```
 
-#### Logging
+### Logging
+
 Production logs can be found in Amazon CloudWatch under the following log groups:
 
-[production/alma_user_loader/sis](https://us-west-1.console.aws.amazon.com/cloudwatch/home?region=us-west-1#logsV2:log-groups/log-group/production$252Falma_user_loader$252Fsis)
+[production/alma_user_loader/sis](https://us-west-1.console.aws.amazon.com/cloudwatch/home?region=us-west-1#logsV2:log-groups/log-group/production$252Falma_user_loader$252Fsis)  
 [production/alma_user_loader/ucpath](https://us-west-1.console.aws.amazon.com/cloudwatch/home?region=us-west-1#logsV2:log-groups/log-group/production$252Falma_user_loader$252Fucpath)
 
 You'll need to sign in with the IAM account alias uc-berkeley-library-it
 and then with your IAM user name and password (created by the DevOps team).
 
----
+In the event Alma-User-Load fails to run:  
+SIS: Because SIS pull the full collection of users for the current semester every run, there are no special steps for recovery. You can either let it run 
+
+
 ## Development
-#### Configuring
-To run locally, clone the repo, then add a .env file to the root of the project. Check `config>secrets.yml` for necessary UCPath, SIS and LDAP settings.
 
-#### Running
-Note - you cannot run students and employees at the same time, you can only run them one at a time.
+### Configuring
+
+To run locally, clone the repo, then add a .env file to the root of the project. (API info can be found in LastPass)  
 
-You can run this directly or using Docker
+> \# UCPath Secrets:  
+> UCPATH_API_URL=https://gateway.api.berkeley.edu/hr/v3/  
+> UCPATH_API_ID=[API ID]  
+> UCPATH_API_KEY=[API KEY]  
+>  
+> \# SIS Secrets:  
+> SIS_API_URL=https://gateway.api.berkeley.edu/sis/v2/students  
+> SIS_API_ID=[API ID]
+> SIS_API_KEY=[API KEY]  
+>   
+> \# LDAP Secrets:  
+> LDAP_HOST=ldap.berkeley.edu  
+> LDAP_PASS=[LDAP Password]
+
+
+
+### Running
+
+Note - you cannot run students and employees at the same time, you can only run them one at a time.  
+Alma-User-Load can be run directly or using Docker.
+
+### Run locally without Docker:
 
-##### Run locally without Docker:
 ```sh
 ruby alma_user_load.rb -t <ucpath|sis>
 ```
 
-##### Run with Docker:
+### Run with Docker:
+
 Build the container:
 ```sh
 docker build -t alma-user-load .
@@ -77,6 +113,17 @@ docker build -t alma-user-load .
   ruby alma_user_load.rb -t ucpath -u '10671129,10674086'
   ```
 
+### Run the test suite:
+```sh
+> rspec
+```
+
+Note - to run the test suite you need to change the LDAP setting in your `.env` file
+> \# For testing:  
+> LDAP_HOST=ldap.fake.edu  
+> LDAP_PASS=FAKEPW  
+
+
 **Commandline Options**
 * --type [ucpath|sis] *required*
 * --term [term-code]  *SIS only*
@@ -85,13 +132,12 @@ docker build -t alma-user-load .
 * --outdir Set the output directory for xml/zip files
 * --help
 
-___
 
-##### Future Improvements:
-- Get "current term" dynamically from API (awaiting access to API)
-- Replace fixtures w/some sort of factory (factorybot?)
+## Future Improvements:
+- Replace fixtures w/some sort of factory (VCR)
 - DRY things up (UCPath vs. SIS --> phone, email, address, names, etc...)
 - SIS - add run by user id (similar to how I setup ucpath)
 - Move 'create_user_record' from user.rb to a separate class
 - Add a check to make sure all necessary config settings are set!
-- Add run by campus-uid to ucpath
+- Fix LDAP tests to override the 
+- Setup a rake command to run the test suite