-
Notifications
You must be signed in to change notification settings - Fork 7
Za/data access #662
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Za/data access #662
Changes from all commits
Commits
Show all changes
108 commits
Select commit
Hold shift + click to select a range
3b5c07d
data access skeleton
amrc-za de4141b
data-access service deployment files
amrc-za ff04d56
js-rx-client - new method to watch acl with permission
amrc-za 938fe05
dataflow start
amrc-za 01167bb
gitignore
amrc-za 7e95851
acs-config-db package.lock update
amrc-za 0956670
data access helm
amrc-za b767419
node modules dockerignored
amrc-za d127366
deploy data access
amrc-za f7bf6f9
data access fixed keytabs/server not found
amrc-za 0e05293
data access - dataflow build metadata and code clean up
amrc-za 5aae2d3
deploy - fixed git krbs keytabs not being created
amrc-za 029b5ea
data access dataflow - keeping track of datasets with contents not ju…
amrc-za 712ffe8
data access get /metadata api ready
amrc-za a927e76
data access api created helper method to get list dataset uuids
amrc-za 7614a40
data access dataflow added method to get dataset content by uuid
amrc-za 203e762
data access api structure_uuid
amrc-za 54e4322
data-access space format and comments
amrc-za 0f338a2
data access api method name change
amrc-za 8b1ef9d
data access api.js - passing configDB instance to apiV1
amrc-za a0e4695
data access - constants - removed structure dataset type uuids
amrc-za 86915cb
data access - flattened dataset definitions map to keys to be dataset…
amrc-za 56b663d
data access api-v1 implemented put structure
amrc-za 48938b2
data access - package json added influx client lib
amrc-za 2441a25
data access - added dataset validity enum
amrc-za ef6a8fa
data access api and dataflow update to keep track of invalid datasets
amrc-za f5d48b0
data access - get data from influx partial implementation
amrc-za 8d0950a
deploy data access - added influx vars and secret
amrc-za cbf9a19
data access updated flux query
amrc-za 9a15d13
data access dataflow watching general info app
amrc-za cbefea7
data access meta uuid endpoint partially implemented
amrc-za 226452e
data access dataflow watching dataset metadata members and their configs
amrc-za c4901d7
lib js-service-client - added DataAccess interface
amrc-za 1dafc51
data access dataflow - changed metadata structure and watching functi…
amrc-za 22cc433
data access dataset metadata endpoint returning function
amrc-za 9d2b1db
data access data endpoint added header with filename
amrc-za 0fc0ee7
data access service-client interface metadata and download methods
amrc-za 829b74a
data access api minor fixes
amrc-za d8f11ac
data access fixed bug with accessing imap items
amrc-za befaa10
data access interface for js-service-client implemented
amrc-za 59db942
data access implemented download endpoint and fixed minor bugs
amrc-za d71d562
data access put influx logic to separate script
amrc-za f1b5d3e
data access influx reader fixed typo
amrc-za 4895101
data access api fixed bug
amrc-za eb506ef
data-access api adding subclass relationship on create/update
amrc-za 7237821
data access dataflow keeping track of dataset members subclasses as p…
amrc-za 32202b8
data access api returning parts as part of metadata
amrc-za 4e77482
data access api checking for read perm for parts metadata
amrc-za 24ad084
data access api parts minor changes
amrc-za 92875d0
data access metadata endpoint fixed dates
amrc-za d7a1c54
data access dataflow passing fplus props separately
amrc-za 0fac854
data access api fixed subclass linking logic
amrc-za 031f6ba
data access api metadata removed fail if general info not found
amrc-za f2bd7e7
data access removed debugging logs
amrc-za 0714734
data access convert to csv fixed char escaping
amrc-za aa606c7
data access influx reader fixed range bug
amrc-za e95dba2
data access api - moved invalid dataset logic to dataflow
amrc-za 5477488
data access api structure create and update logic changed
amrc-za ebe72d8
data access api PUT structure added ServiceError check
amrc-za 8ad04db
data service service client - added status to error throw
amrc-za 92de9c4
data access api removed code duplication for creating subclasses
amrc-za 2e3a27a
data access dataflow refactored build metadata and datasets
amrc-za 7e3592d
data access - streaming data download
amrc-za 76611fd
data access - refactored removed datasetvalidity check
amrc-za 81c221b
rx-client lib added data-access interface - partially implemented
amrc-za eb3f95d
data access metadata uuid api fixed bug with info
amrc-za 446519a
data access notify service added implementations
amrc-za a713d71
rx-client interface for data access added all methods
amrc-za 5413111
data access installed deep-equal
amrc-za c972ba3
data access - fixed notify metadata SEARCH
amrc-za bc7df95
data access structure uuid - removed from and to fields from returnin…
amrc-za 7fc2664
data access package-lock
amrc-za 3a1afc5
data-access notify structure SEARCH done
amrc-za 7ae3ef2
data access added test script for rx client
amrc-za 00a5d21
data-access download data http checking if write is complete
amrc-za 1fa51da
data access added simple test http endpoints via service client
amrc-za 2395d80
js-service-client updated data access interface
amrc-za 5a3eb76
data access package lock
amrc-za 100c738
data access download http endpoint returns zip with separate csv file…
amrc-za f04400a
data access js service client download_data handles zip
amrc-za d251bf7
data access indlux query removed unnecessary cols
amrc-za 3d3bb27
data access removed unnecessary comment
amrc-za 4b09b9a
data access GET /metadata vitest
amrc-za cf2822e
data access http GET /metadata tests - fixed fplus client cache bug
amrc-za 5e2ffba
data access tests GET /metadata added parent child invalid tests
amrc-za fa44cde
data access tests GET /metadata added invalid session child test
amrc-za d074d3d
data access tests GET /metadata removed cache
amrc-za 269c373
data access tests GET /structure
amrc-za 38df7ca
data access webapi added config validation
amrc-za 15774d1
data-access tests structure creaate
amrc-za ee9e0d3
data access tests temp data
amrc-za b2c9d93
data access added delete endpoint
amrc-za 49dfd35
data access test create added subclass relationship tests
amrc-za f54cbce
data access tests update
amrc-za b941535
data access test scripts
amrc-za ceb6a89
data access client lib updated empty return formats
amrc-za d3275e5
ConfigDB removed accidentally pushed package-lock.json
amrc-za d9ef127
Data Access removed accidentally pushed package-lock.json
amrc-za f3023ca
OPCUA-Server-Edge removed .dockerignore that has been accidentally cr…
amrc-za 21a980c
deploy services.yaml returned openid
amrc-za 5810d8b
data-access delete endpoint fixed TypeError
amrc-za 6f3bbec
data access fixed another TypeError
amrc-za dfe264a
data access api fixed TypeError
amrc-za 02131e8
data access updated Express version
amrc-za 1562b6e
data access client library dynamically importing node libraries
amrc-za 9c9ae0a
data access tests added more tests
amrc-za 3d7123b
data access tests download
amrc-za b23f039
data access passing measurement to download
amrc-za File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| node_modules/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| .env | ||
| node_modules/ | ||
| tmp/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| ARG base_version | ||
| ARG base_prefix=ghcr.io/amrc-factoryplus/acs-base | ||
|
|
||
| FROM ${base_prefix}-js-build:${base_version} AS build | ||
| ARG acs_npm=NO | ||
|
|
||
| # Install the node application on the build container where we can | ||
| # compile the native modules. | ||
| RUN install -d -o node -g node /home/node/app \ | ||
| && mkdir /home/node/lib | ||
| COPY --from=lib . /home/node/lib/ | ||
|
|
||
| WORKDIR /home/node/app | ||
| USER node | ||
| COPY package*.json ./ | ||
| RUN <<'SHELL' | ||
| touch /home/node/.npmrc | ||
| npm install --save=false --install-links | ||
| SHELL | ||
| COPY --chown=node . . | ||
| RUN <<'SHELL' | ||
| echo "export const GIT_VERSION=\"$revision\";" > ./lib/git-version.js | ||
| SHELL | ||
|
|
||
| FROM ${base_prefix}-js-run:${base_version} | ||
|
|
||
| # Copy across from the build container. | ||
| WORKDIR /home/node/app | ||
| COPY --from=build --chown=root:root /home/node/app ./ | ||
|
|
||
| USER node | ||
| CMD node bin/api.js |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| top=.. | ||
| include ${top}/mk/acs.init.mk | ||
|
|
||
| repo?=acs-data-access | ||
| k8s.deployment?=data-access | ||
|
|
||
| include ${mk}/acs.js.mk |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| #!/usr/bin/env node | ||
|
|
||
| /* | ||
| * ACS Data Access Service | ||
| * Entry point: api.js | ||
| */ | ||
|
|
||
| import {InfluxDB, flux} from '@influxdata/influxdb-client'; | ||
|
|
||
| import { RxClient, UUIDs } from '@amrc-factoryplus/rx-client'; | ||
| import { WebAPI } from '@amrc-factoryplus/service-api'; | ||
| import {DataFlow} from '../lib/dataflow.js'; | ||
| import { APIv1 } from '../lib/api-v1.js'; | ||
| import {DataAccessNotify} from '../lib/notify.js'; | ||
| import { InfluxReader } from '../lib/influx-reader.js'; | ||
|
|
||
| const { env } = process; | ||
|
|
||
| const fplus = await new RxClient({ | ||
| env, | ||
| }).init(); | ||
|
|
||
| const Version = "2.0.0"; | ||
|
|
||
| const debug = fplus.debug; | ||
| debug.log("app", "Starting acs-data-service revision %s"); | ||
|
|
||
| const influxClient = new InfluxDB({ | ||
| url: env.INFLUXDB_URL, | ||
| token: env.INFLUXDB_TOKEN | ||
| }); | ||
|
|
||
|
|
||
| // the dataflow object is full of sequences that does most of the work of this service. Sequence means an RX observable. | ||
| const data = new DataFlow({ | ||
| debug, | ||
| cdb: fplus.ConfigDB, | ||
| auth: fplus.Auth, | ||
| }); | ||
|
|
||
| const influxReader = new InfluxReader({ | ||
| debug, | ||
| influx_client: influxClient, | ||
| influx_org: env.INFLUXDB_ORG, | ||
| influx_bucket: env.INFLUXDB_BUCKET, | ||
| }) | ||
|
|
||
| const apiv1 = new APIv1({ | ||
| data, | ||
| debug, | ||
| auth: fplus.Auth, | ||
| cdb: fplus.ConfigDB, | ||
| influxReader | ||
| }); | ||
|
|
||
| const api = await new WebAPI({ | ||
| debug: fplus.debug, | ||
| ping: { | ||
| version: Version, | ||
| service: UUIDs.Service.DataAccess, | ||
| software: { | ||
| vendor: 'AMRC', | ||
| application: 'acs-data-access', | ||
| }, | ||
| }, | ||
| realm: env.REALM, | ||
| hostname: env.HOSTNAME, | ||
| keytab: env.SERVER_KEYTAB, | ||
| http_port: env.PORT, | ||
| max_age: env.CACHE_MAX_AGE, | ||
| routes: app => { | ||
| app.use("/v1", apiv1.routes); | ||
| } | ||
|
|
||
| }).init(); | ||
|
|
||
|
|
||
| const notify = new DataAccessNotify({ | ||
| api, data, debug, | ||
| auth: fplus.Auth | ||
| }); | ||
|
|
||
| debug.log("app", "Running Data Access DataFlow"); | ||
| data.run(); | ||
|
|
||
| debug.log("app", "Running Data Access Notify"); | ||
| notify.run(); | ||
|
|
||
| debug.log("app", "Running Data Access WebAPI") | ||
| api.run(); | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.