prepare api for oc3-frontend - #157
Merged
Merged
Conversation
the node and service names on the alert endpoints
The directory holds per-machine tool settings, which do not belong in the repository. The rule comes last, after the whitelist entries: the JSON exception would otherwise bring its settings file back.
JSONProblemf returns nil once the problem response is written. The require* guards and the service resolvers returned its result, so their callers went on with the action after answering 401, 403 or 404. They now go through denyRequest, which returns errRequestDenied; echo leaves the already committed response untouched. resolveService also selects svc_id: the query builder refuses a query without selected columns, and the existence check failed with a 500.
A log event names its node and service by id only. GET /logs and
GET /logs/{log_id} now LEFT JOIN nodes and services, so that the
nodes.nodename and services.svcname props can be selected; most events
name neither, hence the outer joins. The responses are typed as
LogListResponse instead of the generic ListResponse.
An instance row carries only the service and node ids. The svcmon mapping joins nodes and services, nodes through a LEFT JOIN so that an instance whose node row is missing is not dropped. orderby resolves a "table.column" prop through the mapping joins, as props selection does, so that instances can be sorted by service or node name. The instance responses are typed as InstanceListResponse.
A node names its cluster by id only. The node mapping joins the clusters table, through a LEFT JOIN since a node may name no cluster or one the collector does not know, and NodeRow gains clusters.cluster_name.
Dates are validated as YYYY-MM-DD or YYYY-MM-DD hh:mm:ss, and an empty date clears the setting instead of being rejected by the datetime column. The nodes take the setting's dates by a copy inside the database: a date read back through the driver came in RFC 3339 form, which a datetime column refuses in strict mode. The responses are typed as ObsolescenceSettingListResponse.
POST /filters/{filter_id} validates the resulting definition with the
rules of POST /filters: known table and operator, existing column, no
other filter with the same definition (409). f_label, a generated column,
is refused. The transaction is committed before the filter is read back,
which returned the previous values otherwise. The filter responses are
typed as FilterListResponse.
fset_stats and f_log_op are declared as enums and checked on every endpoint that sets them; a filterset name cannot be emptied nor take the name of another one (409). The transaction is committed before the filterset is read back. The filterset, composition, usage and export responses get their own schemas.
GET /nodes/{node_id}/hardware, hbas, disks, ips and alerts, and
GET /services/{svc_id}/disks, now declare their row schemas instead of
the generic ListResponse.
Declare a network, with the NetworkManager privilege. The network address must match the netmask; begin, end and broadcast are computed by the database, and team_responsible defaults to the caller's primary group. A name or an address already declared is refused with a 409.
Create a user with its private group, with the UserManager privilege. An email or username already in use is refused with a 409. The password is hashed in the web2py format the sign-in already verifies; without one, the account exists but cannot sign in.
A queued action names its node, and its service when it has one, by id only. GET /actions LEFT JOINs nodes and services so that their names can be selected, and the access check qualifies node_id, now ambiguous.
Queue an agent action on a node, as the historical collector action menu did: inventory pushes, checks, sysreport, scanscsi, freeze and thaw. The command is queued bare for a pull or feed node and wrapped in an ssh call for a push node, reaching the node by its connect_to or best routable address. Requires the NodeExec privilege and responsibility for the node.
POST /services/{svc_id}/actions queues an action for a node of the
service seen alive in the last 15 minutes, acting on the whole service.
POST /services/{svc_id}/instances/{node_id}/actions acts on that instance
only, with --local for agents that understand it. Accepted actions are
push resinfo, push config, freeze and thaw. Requires the NodeExec
privilege and responsibility for the service.
GET /nodes, /services, /services_instances and /actions, and the filterset node and service lists, accept a repeated filter=prop:expr parameter, combined with AND: a substring by default, ~regex, in:a,b, eq/ne/gt/gte/ lt/lte comparisons, empty and !empty. Props resolve like orderby, joined names included, and the query builder adds the join of a filtered column even when it is not selected. An unknown prop or an invalid regular expression is refused with a 400.
Replace the dedicated POST /nodes/{node_id}/actions,
POST /services/{svc_id}/actions and
POST /services/{svc_id}/instances/{node_id}/actions endpoints with
PUT /actions, as the historical collector's REST API and action menu do
(rest_put_action_queue, json_action_one). The body is one entry or a
list; node_id and svc_id queue an instance action, svc_id alone a service
action from a live node, node_id alone a node action, and vmname names
the node. rid limits an instance action to resources, entries differing
only by rid being merged into one action. One entry answers with the
queued action, a list with the queued actions and the refusals.
Instance actions follow do_instance_action: NodeExec privilege and
responsibility for the service, the node only having to exist. The agent
version is read past a leading "v", so that om3 agents get --local and an
instance freeze no longer freezes the whole object.
List endpoints fill meta.total whenever meta is requested, the default, as the historical collector did. The total comes from the page when it tells it (no limit, partial page, empty first page); otherwise the fetcher runs again with the same access control, filters and grouping, selecting COUNT(*) OVER () without sort for one row, so that a grouped list counts its groups. GET /alerts and GET /tags return it too. A failed count is logged and the total left out.
The sysreports history is committed with git, and so is the history of the form definitions: the image only had bash, so neither could work.
Port the forms API of the historical collector (api_forms.py, the forms
part of api_workflows.py and lib_forms.py), keeping its behavior:
- /forms and /forms/{form_id}: list, show, create, modify and delete,
with the FormsManager privilege, the publication and responsibility
checks and the collector's messages. The body may be one entry or a
list, JSON or form-encoded. form_definition is form_yaml parsed.
- /forms/{form_id}/publications and /responsibles, and the bulk
/forms_publications and /forms_responsibles.
- /forms/{form_id}/am_i_responsible, answering {"data": bool}.
- /forms/{form_id}/revisions, /revisions/{cid}, /diff/{cid} and
/rollback/{cid}, over one git repository per form under
server.directories.forms.
- PUT /forms/{form_id}: submission. The data is validated against the
form inputs (conditions, forced keys, strict static and dynamic
candidates), a results record is stored, and the outputs run: db,
rest (calling this API as the submitter, or an external url, with the
vm2 mangler), script, mail (server.mail.*) and workflow. Async forms
answer at once and run in the background. Internal forms, the
negative ids, are embedded.
- /form_output_results/{results_id} (GET, PUT), /forms_revisions and
/forms_store, with /forms_store/{store_id}/dump.
List endpoints may declare virtual props, computed after the fetch
from the props they require, as the collector's vprops.
List the packages installed on the nodes the user can see, as the historical packages table: joined with nodes, whose app scopes the access, and left-joined with pkg_sig_provider for the provider of the signing key (sig_provider). Ordered by node name, package name and architecture by default; the nodes. props come through the join, and the list takes the filter parameter and returns its total. The response is typed in the spec (PackageRow, PackageListResponse).
List the requests: the workflows started by the submission of a form with a workflow output, as the historical rest_get_workflows, to any authenticated user. The workflows query left-joins forms_revisions by form_md5, so that the workflow mapping also offers form_name, form_folder and form_yaml; they stay out of the default props, which keeps the /forms_store dump output unchanged. The list takes the filter parameter, and its response is typed (WorkflowRow, WorkflowListResponse).
The assigned parameter keeps the pending workflows related to the caller's team, their full name and the roles of their non-privilege groups, as the forced filters of the historical tables: "team" those whose last assignee is in the team (Assigned to my team), "tiers" those created by the team and assigned to someone else (Pending tiers action). An unknown value is a 400.
formError, formErrorf, formProblem and formInternal carry an HTTP status through the handlers and write it once, as the historical collector raised HTTP(status, message). They are not specific to forms: renamed httpError, httpErrorf, httpProblem and httpInternal, they move to problems.go for the compliance handlers to use as well.
Port the /compliance/logs and /compliance/status endpoints of the
historical api_compliance: GET /compliance/logs and /logs/{log_id}, GET
/compliance/status and /status/{status_id}, scoped as q_filter(node_field)
to the nodes whose app is published to one of the caller's groups; DELETE
/compliance/status/{status_id} and its bulk form DELETE /compliance/status
({"id": ...}), which require the CompExec privilege and the node's
responsibility, and log the deletion.
The run and log rows are typed (ComplianceLogRow and its list response;
ComplianceStatusListResponse). The fset-id filterset of the historical
lists is not supported, as nowhere else in oc3; nor is the refresh of the
"compliance differences in cluster" dashboard alert, which oc3 does not
compute.
Port the ruleset endpoints of the historical api_compliance:
- GET /compliance/rulesets and /rulesets/{rset_id} (id or name), the
rulesets published to one of the caller's groups;
- POST /compliance/rulesets, creating a ruleset published to and under
the responsibility of the caller's default group (the Manager group
without one), or updating it when the name exists;
- POST /compliance/rulesets/{rset_id}, updating its name, type or
publicity, checked;
- PUT /compliance/rulesets/{rset_id} action=clone, copying its variables,
children and, for a contextual ruleset, its filterset as <name>_clone;
- DELETE /compliance/rulesets/{rset_id} and the bulk DELETE
/compliance/rulesets ({"id": ...}), removing its relations;
- GET /compliance/rulesets/{rset_id}/usage and /am_i_responsible.
Changes require the CompManager privilege and, but for creation, the
responsibility of the ruleset; refusals are 403, unknown rulesets 404,
taken names 409. Every change is logged, and creation, renaming, cloning
and deletion rebuild comp_rulesets_chains, as comp_rulesets_chains().
Bodies may be JSON or form-encoded, as the historical examples post.
Port the ruleset variable endpoints of the historical api_compliance:
- GET /compliance/rulesets/{rset_id}/variables and /variables/{var_id}
(id or name), for a ruleset published to the caller, to one of their
groups or to Everybody;
- POST /compliance/rulesets/{rset_id}/variables and its form naming the
ruleset in the body, POST /compliance/rulesets_variables, creating a
variable or updating the one of that name;
- POST /compliance/rulesets/{rset_id}/variables/{var_id}, updating its
name, class or value; a name taken in the ruleset is a 409;
- PUT /compliance/rulesets/{rset_id}/variables/{var_id} action=copy or
move to dst_ruleset;
- DELETE /compliance/rulesets/{rset_id}/variables/{var_id}.
Changes require the CompManager privilege and the responsibility of the
ruleset (for a copy, the publication of the source), record the author
and the date, and are logged. var_value may be any JSON value, stored
serialized. Two departures from the historical handlers: the deletion
checks the privilege and the responsibility, as its description says,
where the code only checked the publication; and a copy or a move to a
ruleset already holding a variable of that name is refused.
The ruleset_variable props mapping and the RulesetVariableRow schema
describe the rows.
Port the ruleset hierarchy and filterset endpoints of the historical
api_compliance:
- POST and DELETE /compliance/rulesets/{rset_id}/rulesets/{child_rset_id},
as attach_ruleset_to_ruleset() and detach_ruleset_from_ruleset(): by a
CompManager responsible for the parent, the child being published to
them for a detach. Attaching a ruleset to itself (400), twice or under
one of its descendants (409) is refused; the ruleset chains are rebuilt.
- POST and DELETE /compliance/rulesets/{rset_id}/filtersets/{fset_id}, the
filterset of a contextual ruleset, replaced on attach and removed
whichever it is on detach, as the historical handlers do.
Rulesets and filtersets are given by id or name; changes are logged.
Port the ruleset group endpoints of the historical api_compliance:
- GET /compliance/rulesets/{rset_id}/publications and /responsibles, the
groups of a ruleset, a non-manager seeing only their own among them;
- POST and DELETE /compliance/rulesets/{rset_id}/publications/{group_id}
and /responsibles/{group_id}, and their bulk forms naming both in the
body, /compliance/rulesets_publications and /rulesets_responsibles.
As attach_group_to_ruleset() and detach_group_from_ruleset(), changes
require the CompManager privilege and the responsibility of the ruleset;
a non-manager may attach only one of their groups; Everybody may not be
made responsible, nor be given the publication of a contextual ruleset;
an already attached group is answered with an information. Groups are
given by id or role; changes are logged.
The group logic is written once over a CompKind naming the tables of a
kind, for the modulesets to use as well.
comp_rulesets_services.slave and comp_modulesets_services.slave are varchar(1) columns the historical collector fills with "T" or "F", as its web2py boolean fields do. oc3 passed a Go boolean, stored as "1" or "0", which the historical collector reads as false; and compared the column with a boolean, which MySQL casts "T" and "F" to 0 alike, so that the non-encapsulated queries also matched encapsulated attachments. Attachments are now stored "T" or "F"; queries read "T" and "1" as true, anything else as false, so that rows stored earlier keep their meaning.
Port the ruleset attachment endpoints of the historical api_compliance:
- GET /compliance/rulesets/{rset_id}/nodes and /services (slave for the
encapsulated services), what a ruleset published to the caller is
attached to, among the nodes and services the caller may see;
- GET /compliance/rulesets/{rset_id}/candidate_nodes and
/candidate_services, what it may be attached to: the nodes whose
responsible team, the services whose app has a responsible group, among
the ruleset's publication groups, not attached yet;
- POST and DELETE /compliance/rulesets_nodes and /rulesets_services, the
bulk forms naming the ruleset (ruleset_id or ruleset_name), the node or
the service and the slave flag (encap accepted) in the body, handled as
the existing /nodes/{node_id}/compliance/rulesets/{rset_id} and
/services/{svc_id}/compliance/rulesets/{rset_id}.
The list queries are written over the CompKind, which names the node and
service attachment tables too, for the modulesets to use as well.
Port the moduleset endpoints of the historical api_compliance:
- GET /compliance/modulesets and /modulesets/{modset_id} (id or name),
the modulesets published to one of the caller's groups;
- POST /compliance/modulesets, creating a moduleset authored by the
caller, published to and under the responsibility of their default
group; a taken name is a 409;
- POST /compliance/modulesets/{modset_id}, renaming it and recording the
update date;
- PUT /compliance/modulesets/{modset_id} action=clone, copying its
modules, rulesets and children as <name>_clone;
- DELETE /compliance/modulesets/{modset_id} and the bulk DELETE
/compliance/modulesets ({"id": ...}), removing its relations;
- GET /compliance/modulesets/{modset_id}/usage (the parent modulesets)
and /am_i_responsible.
Changes require the CompManager privilege and, but for creation, the
responsibility of the moduleset; they are logged. The visibility and the
publication checks, Everybody included, are written over the CompKind.
List, show, create, update and delete the modules of a moduleset, as the historical collector's /compliance/modulesets/<id>/modules handlers. A module name is unique in its moduleset, the autofix flag is stored as T or F, and changes require the CompManager privilege and the responsibility of the moduleset.
Attach and detach child modulesets and rulesets of a moduleset, as the historical /compliance/modulesets/<id>/modulesets/<id> and /compliance/modulesets/<id>/rulesets/<id> handlers. Unlike the historical collector, an attachment closing a moduleset loop is refused, and the attached moduleset or ruleset must be published to the caller. The loop walk is shared with the ruleset composition.
The standard library parses the form body of POST, PUT and PATCH requests only, so a DELETE sent with curl -d, as the historical examples of the bulk detach endpoints do, reached the handlers without its keys.
List, attach and detach the publication and responsible groups of a moduleset, by path or with the bulk /compliance/modulesets_publications and /compliance/modulesets_responsibles forms, sharing the ruleset rules: a CompManager responsible for the moduleset may attach only their own groups, and Everybody may not be made responsible.
List the nodes and services a moduleset is attached to and its candidates, and attach or detach a moduleset named in the body with the bulk /compliance/modulesets_nodes and /compliance/modulesets_services forms, which delegate to the node and service compliance handlers.
Export rulesets and modulesets, one or all those published to the
caller, with their descendants and the filtersets and rulesets they
use, and import such an export, as the historical /compliance/import
and /compliance/{rulesets,modulesets}/export handlers.
Objects existing by name are reused and the missing relations added.
Unlike the historical import, it requires the CompManager privilege,
adds to an existing ruleset or moduleset only when the caller is
responsible for it, refuses relations closing a loop, and runs in a
transaction so that a refused import leaves nothing behind.
GET /compliance/modulesets_modules lists, as the historical v_comp_modulesets view, one row per module with its moduleset and the moduleset's responsible and publication teams; a moduleset without module is listed on a row of its own, with a zero module id. The rows are restricted to the modulesets published to the caller, all of them for a manager. The teams come from derived tables whose columns are declared in cdb, so they can be filtered and sorted like the other props.
GET /compliance/rulesets_variables lists, as the historical v_comp_rulesets view, the variables of each ruleset visible to the caller: its own, then those of each ruleset it encapsulates, found through comp_rulesets_chains and named by encap_rset and chain, with the ruleset's filterset and its responsible and publication teams. A ruleset of the chain without variable is listed on a row of its own, with a zero variable id. The team join of the modulesets list becomes compTeamsJoin, shared by both lists through their CompKind.
…rset GET /compliance/logs joins the nodes and services tables, so the nodes.* and services.* props give the names of a run's node and service, and accepts fset_id, a filterset id or name restricting the runs to the nodes it selects, as apply_filters_id() does on the node of a run. An unknown filterset is a 404.
POST /users/self/password takes the current password, checked as at sign in, and a new one of at least 8 characters, different from the current one. The new password is stored as a web2py hash, accepted by both collectors, and the change is logged without it. No privilege is required: knowing the current password is the proof.
GET /networks lists the declared networks to any authenticated user, as
the historical collector did.
POST /networks/{net_id}/segments ports rest_post_network_segments: it
requires the NetworkManager privilege and, unless the caller is a
manager, membership of the network's responsible team. The range must be
IPv4, lie inside the network and not overlap another segment of it
(409). The caller's primary group becomes responsible for the segment,
and the creation is logged as networks.segment.create.
Each prop of the node_ip mapping keeps its expression and now names the v_nodenetworks column it reads, so orderby, groupby and filter accept them. GET /ips declares the filter parameter and applies its conditions.
POST /users/self/password refuses a new password shorter than 12 characters, counted in characters rather than bytes. User creation keeps its own minimum.
GET /search looks for the text in 14 kinds of objects, as the historical collector's search did: nodes, services, instances, applications, node addresses, disks, tags, users, teams, requests, modulesets, rulesets, filtersets and forms. Each kind is read through its own list and access control, with a case-insensitive "contains" match on its identifying props, all kinds at once. A request also matches by its number, private user groups are left out of the teams. Groups come with more set when other objects match beyond the limit, and a kind that fails reports it without failing the others. The Apps, Disks, Users, Groups and Filtersets lists now apply the column filters they were given and declare the filter parameter.
The om3 agents inventory the SAN switches their configuration declares, and fed them to the old collector only, through its update_brocade rpc, which stored the command outputs and parsed them into the switches, san_zone and san_zone_alias tables. oc3 had no feed for it, and will drop the rpc api. The feeder serves POST /sanswitch, taking the switch name, its type and the raw output of each command, by name: switchshow, nsshow and zoneshow for a brocade. The report is keyed by switch, as <type>@<name>, and not by node: a couple of nodes of the whole infrastructure inventory the switches, and the latest report of a switch is the one to parse. The sanSwitch job, on the san_switch queue, parses the outputs with a port of the old collector parser, quirks included, and replaces the rows of the switch, of the aliases of each of its configurations, and of the zones of its effective one: the rows are upserted on the unique keys of the tables, and the older rows deleted. A switchshow without its port table is refused rather than read as a switch with no port, which would delete them all. A deployment serves the feed once a worker lists san_switch in its queues.
…ions-only reports The members of a sysreport archive and the deleted files were joined to the tree of the node as given: a name climbing with ".." wrote or removed files out of it. Every path is now rooted at the tree of the node, where a ".." stops, and only the regular files of the archive are written, each closed once written rather than at the end of the request. The archive is optional: a report of the deletions of a node has none, and was refused. A full report, the full field set, holds every file and command output of the node: the files the collector holds for the node and the archive does not are removed, the git history of the tree left alone. It replaces the listing of what the collector holds, which the agent asked the old collector for to compute the deletions of a forced report.
POST /users/{user_id} changes the first name, last name or email of the
signed-in user, as the profile form of the historical collector allows:
only the caller's own account, without privilege. Names are trimmed and
at most 128 characters; the email must be a valid address, at most 512
characters and not used by another user. The change is logged with the
fields changed, and the old and new email.
POST /users/{user_id} now changes another user's first name, last name
or email when the caller has the UserManager privilege, or is a Manager,
as rest_post_user did. Other callers are refused with a 403, an unknown
user with a 404.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.