Resolve DEV-46 "Implement lbnl sftp handler"

Author: danschmidt5189

.gitignore

@@ -1,6 +1,11 @@
 # secrets file
 .env
 
+# rubocop creates this
+.cache
+
+.*_history
+
 vendor
 .bundle
 

.rubocop.yml

@@ -5,6 +5,7 @@ AllCops:
     - 'bin/**/*'
     - 'db/**/*'
     - 'out/**/*'
+    - 'spec/**/*'
 
 # Allow one line around block body (Layout/EmptyLines will still disallow two or more)
 Layout/EmptyLinesAroundBlockBody:
@@ -34,6 +35,9 @@ Lint/SuppressedException:
 Layout:
   Max: 150
 
+Metrics/MethodLength:
+  Max: 50
+
 # Confusing and weird
 Naming/VariableNumber:
   Enabled: False

Dockerfile

@@ -36,7 +36,7 @@ USER $APP_USER
 
 # Uses the get_gobi script as the entrypoint, so any arguments passed to `docker run`
 # at invocation are passed directly to this script.
-ENTRYPOINT ["ruby","/opt/app/lib/get_gobi.rb"]
+ENTRYPOINT ["/opt/app/bin/berkeley_library-sftp_handler.rb"]
 
 # =============================================================================
 # Target: development

Gemfile

@@ -1,13 +1,16 @@
 source 'https://rubygems.org'
 
-gem 'colorize'
-gem 'erb'
-gem 'getoptlong'
+gem 'chronic', '~> 0.10.2'
 gem 'net-sftp'
-gem 'rspec'
-gem 'rspec_junit_formatter'
-gem 'rubocop', require: false
-gem 'rubocop-checkstyle_formatter', require: false
 gem 'thor', '~> 1.1'
-gem 'uri'
-gem 'webmock'
+
+group :test do
+  gem 'colorize'
+  gem 'rspec'
+  gem 'rspec-its', '~> 1.3'
+  gem 'rspec_junit_formatter'
+  gem 'rubocop', require: false
+  gem 'rubocop-checkstyle_formatter', require: false
+  gem 'timecop', '~> 0.9.5'
+  gem 'webmock'
+end

Gemfile.lock

@@ -4,14 +4,11 @@ GEM
     addressable (2.8.0)
       public_suffix (>= 2.0.2, < 5.0)
     ast (2.4.2)
-    cgi (0.3.2)
+    chronic (0.10.2)
     colorize (0.8.1)
     crack (0.4.5)
       rexml
     diff-lcs (1.5.0)
-    erb (2.2.3)
-      cgi
-    getoptlong (0.1.1)
     hashdiff (1.0.1)
     net-sftp (3.0.0)
       net-ssh (>= 5.0.0, < 7.0.0)
@@ -32,6 +29,9 @@ GEM
     rspec-expectations (3.11.0)
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.11.0)
+    rspec-its (1.3.0)
+      rspec-core (>= 3.0.0)
+      rspec-expectations (>= 3.0.0)
     rspec-mocks (3.11.0)
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.11.0)
@@ -53,8 +53,8 @@ GEM
       rubocop (>= 1.14.0)
     ruby-progressbar (1.11.0)
     thor (1.2.1)
+    timecop (0.9.5)
     unicode-display_width (2.1.0)
-    uri (0.11.0)
     webmock (3.14.0)
       addressable (>= 2.8.0)
       crack (>= 0.3.2)
@@ -64,16 +64,16 @@ PLATFORMS
   x86_64-linux
 
 DEPENDENCIES
+  chronic (~> 0.10.2)
   colorize
-  erb
-  getoptlong
   net-sftp
   rspec
+  rspec-its (~> 1.3)
   rspec_junit_formatter
   rubocop
   rubocop-checkstyle_formatter
   thor (~> 1.1)
-  uri
+  timecop (~> 0.9.5)
   webmock
 
 BUNDLED WITH

Jenkinsfile

@@ -2,14 +2,13 @@ dockerComposePipeline(
   commands: [
     [
       [run: 'rspec', entrypoint: '/bin/sh -c'],
-      [run: 'rubocop', entrypoint: '/bin/sh -c'],      
+      [run: 'rubocop', entrypoint: '/bin/sh -c'],
     ]
   ],
   artifacts: [
-    junit   : 'artifacts/rspec/*.xml',
-    html    : [
-      'RuboCop'      : 'artifacts/rubocop',
+    junit: 'artifacts/rspec/*.xml',
+    html: [
+      Rubocop: 'artifacts/rubocop',
     ],
   ]
 )
-

README.md

@@ -1,69 +1,58 @@
-# GOBI Processor
+# SFTP Handler
 
-A command-line tool for retrieving GOBI order marc files (.ord) via sftp.
+A command-line tool for SFTP-ing files from different providers. Currently supports LBNL patron files and Gobi E-Order files.
 
-## Running it locally
+## Getting started
 
-To run the application for real, you'll need to configure a local `.env` file with the GOBI password, which you can obtain from LastPass (search for "gobi"). Don't worry: this file is git-ignored.
-
-```ini
-# ./.env (in the directory you cloned this from GitLab)
-GOBI_PASS=password-from-lastpass
-```
-
----
-
-> **Additional Configuration:** The app accepts a few other configuration options, which you can see in `config/connections.yml`. The default values should work fine, however, so you probably don't need to change them. Also note that everything is mocked in the rspec tests, where you're not actually hitting the GOBI server.
-
----
-
-With that configured, you can run the application like so:
-
-```sh
-docker compose build # as always, you need to build it first
-docker compose run --rm app ebook0413.ord
-```
-
-Today's GOBI order file will be at `/gobiord/eboo{mmdd}.ord` on their server. By default, the app copies it into the `/opt/app/data` directory within the container (mapped to `./data` in development), but this can be overwritten by the `--local_dir` flag:
+The application is dockerized, so for starters, just build it:
 
 ```sh
-docker compose run --rm app --local_dir /path/in/container
+docker compose build
 ```
 
-Look for a specific file by passing its name as an argument:
-
-```sh
-docker-compose run --rm app ebook0413.ord
-```
+Besides that, it's a fairly standard Ruby library:
+- [Thor](https://github.com/rails/thor) for the CLI
+- RSpec for testing (along with a few RSpec plugins)
 
-You can also pass additional arguments when looking for a specific file:
+The `bin/berkeley_library-sftp_handler.rb` script is the container's entrypoint, so you can run it ad-hoc like so:
 
 ```sh
-docker-compose run --rm app ebook0413.ord \
-    --remote_dir gobiord-staging \
-    --local_dir path_to_local_dir
+# view help docs
+docker compose run --rm app help
+docker compose run --rm app help gobi
+docker compose run --rm app help lbnl
+
+# run the gobi loader
+docker compose run --rm app gobi
+docker compose run --rm app gobi --filename specific-file.ord
+
+# run the lbnl loader
+docker compose run --rm app lbnl
+docker compose run --rm app lbnl --filename specific-file.zip
 ```
 
-## Testing
+By default, files are downloaded to `./data`. You can override this with the `--local-dir` option. See the specific help output or the [CLI class](lib/berkeley_library/sftp_handler/cli.rb) for details.
 
-The app includes a number of RSpec tests integrated into its Jenkins pipeline. You can run the tests locally via:
+Open a shell (e.g. to run tests) by overriding the entrypoint:
 
 ```sh
-docker compose run --rm --entrypoint=rspec app
+docker compose run --rm --entrypoint bash app
 ```
 
-Or by shelling into the container and running RSpec from there:
+### Secrets and .env
 
-```sh
-docker compose run --rm --entrypoint=sh app
-rspec
-```
+SSH passwords and key data should be stored in the .env file. This file is git-ignored, so don't worry about it being committed. The contents of different secrets will be shared with you via LastPass; just ask for it in Slack.
 
----
+```ini
+LIT_GOBI_PASSWORD="the-password"
+LIT_LBNL_KEY_DATA="-----BEGIN RSA PRIVATE KEY-----
+...
+-----END RSA PRIVATE KEY-----"
+```
 
-> **Insecure KEX:** As of 04/28/22, Gobi's server only supports outdated Diffie-Hellman key exchange algorithms, which you local sftp client will probably (and correctly) refuse. If you want to test manually, then for now you must explicitly allow the old algorithm by passing the following option to your sftp client: `-oKexAlgorithms=+diffie-hellman-group1-sha1`.
+## Adding new downloaders
 
----
+To add a new downloader, create a class under BerkeleyLibrary::SftpHandler::Downloader, inherit from Downloader::Base, and implement the `download!` method. Then, integrate it into the CLI by adding a new method to BerkeleyLibrary::SftpHandler::Cli. See the existing examples for Lbnl and Gobi for more guidance.
 
 ## Production configuration
 

bin/berkeley_library-sftp_handler.rb

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+
+File.join(File.expand_path('..', __dir__), 'lib').tap do |lib|
+  $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+end
+
+require 'docker'
+Docker::Secret.setup_environment!
+
+require 'berkeley_library/sftp_handler'
+
+BerkeleyLibrary::SftpHandler::Cli.start(ARGV)

config/connections.yml

@@ -4,7 +4,8 @@ services:
       context: .
       target: development
     environment:
-      GOBI_PASS: "${GOBI_PASS}"
+      LIT_GOBI_PASSWORD: "${LIT_GOBI_PASSWORD}"
+      LIT_LBNL_KEY_DATA: "${LIT_LBNL_KEY_DATA}"
     image: containers.lib.berkeley.edu/lap/gobi_sftp/development:latest
     init: yes
     volumes: