Skip to content

Commit ff1a81f

Browse files
manonthegithubmanonthegithub
committed
docs update
1 parent 950675b commit ff1a81f

4 files changed

Lines changed: 177 additions & 3 deletions

File tree

‎SUMMARY.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@
3232

3333
* [API](docs/usage/api/README.md)
3434
* [Databus Mods](docs/mods.md)
35-
* [Databus Client](docs/databus-client.md)
35+
* [Databus Client](docs/usage/databus_client.md)
36+
* [Integration with CI (Jenkins)](docs/usage/ci.md)
3637
* [Running Your Own Databus Server](docs/running-your-own-databus-server/running-own-server.md)
3738
* [Run with Docker](docs/running-your-own-databus-server/run-with-docker.md)
3839
* [Configuration](docs/running-your-own-databus-server/configuration.md)

‎docs/databus-client.md‎

Whitespace-only changes.

‎docs/usage/ci.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Integration with CI (Jenkins)
2+
3+
In this article we briefly describe how to use DBpedia Databus with continuous integration systems like [Jenkins](https://www.jenkins.io) and provide several examples of jenkins pipelines.
4+
5+
## Publishing your data files (datasets) into Databus.
6+
You can use CI tools for publishing metadata of your data into Databus.
7+
Here is an example of jenkins pipeline for that:
8+
9+
```groovy
10+
// databus DataID template for publishing (this is a minimal version)
11+
// here we are publishing only one file
12+
def req(downloadUrl, username, artifact, version, licenseUrl){
13+
return """{
14+
"@context": "https://downloads.dbpedia.org/databus/context.jsonld",
15+
"@graph": [
16+
{
17+
"@type": "Version",
18+
"@id": "https://databus.dbpedia.org/${username}/jenkins/${artifact}/${version}",
19+
"hasVersion": "${version}",
20+
"title": "Test jenkins",
21+
"description": "Test jenkins",
22+
"license": "${licenseUrl}",
23+
"distribution": [
24+
{
25+
"@type": "Part",
26+
"formatExtension": "txt",
27+
"compression": "none",
28+
"downloadURL": "${downloadUrl}"
29+
}
30+
]
31+
}
32+
]
33+
}"""
34+
}
35+
36+
pipeline {
37+
agent any
38+
stages {
39+
stage("Generate data"){
40+
steps{
41+
// we create file for demonstration purpose
42+
script {
43+
sh "echo 'Hello World!' > 'jenkins-test-file-${BUILD_DATE}-${BUILD_NUMBER}.txt'"
44+
}
45+
}
46+
}
47+
// we transfer the file to a nginx www location, the file gets downloadable.
48+
stage('SSH transfer') {
49+
steps([$class: 'BapSshPromotionPublisherPlugin']) {
50+
sshPublisher(
51+
continueOnError: false, failOnError: true,
52+
publishers: [
53+
sshPublisherDesc(
54+
configName: "nginx",
55+
verbose: true,
56+
transfers: [
57+
sshTransfer(sourceFiles: "*.txt", remoteDirectory: "jenkins-test/${BUILD_DATE}")
58+
]
59+
)
60+
]
61+
)
62+
}
63+
}
64+
// we publish the file to databus specifying its download link
65+
stage("Publish to Databus"){
66+
steps{
67+
script{
68+
// USERNAME is your Databus username
69+
withCredentials([usernamePassword(credentialsId: 'DBUS-Kikiriki', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD')]){
70+
def body = req(
71+
// download uri
72+
"http://test.dbpedia.org/data/jenkins-test/${BUILD_DATE}/jenkins-test-file-${BUILD_DATE}-${BUILD_NUMBER}.txt",
73+
// your Databus username
74+
USERNAME,
75+
"jenkins",
76+
// you specify this as a Databus version
77+
"${BUILD_DATE}-${BUILD_NUMBER}",
78+
"https://dalicc.net/licenselibrary/Apache-2.0"
79+
)
80+
echo """DataID:
81+
${body}"""
82+
83+
84+
def response = httpRequest validResponseCodes: "200",
85+
consoleLogResponseBody: true,
86+
httpMode: 'POST', quiet: true,
87+
requestBody: body,
88+
url: "https://databus.dbpedia.org/api/publish",
89+
customHeaders:[
90+
// here is you Databus Api Key
91+
[name:'X-API-KEY', value: PASSWORD],
92+
[name: "Content-Type", value: "application/ld+json"]
93+
]
94+
95+
echo "Status: ${response.content}"
96+
}
97+
}
98+
}
99+
}
100+
}
101+
}
102+
```
103+
104+
## Downloading data files (datasets) from Databus.
105+
106+
Here is a sample script of how to download the latest version of an artifact from Databus in a jenkins pipeline:
107+
108+
```groovy
109+
// A template for SPARQL query.
110+
// We query 1 file of the latest version of an artifact.
111+
// !!! NOTE that it queries only one file (LIMIT 1), in our case with 1-file artifact it works
112+
def req(artifact){
113+
return """
114+
PREFIX dcat: <http://www.w3.org/ns/dcat#>
115+
PREFIX databus: <https://dataid.dbpedia.org/databus#>
116+
PREFIX dct: <http://purl.org/dc/terms/>
117+
118+
SELECT ?file WHERE
119+
{
120+
GRAPH ?g
121+
{
122+
?dataset databus:artifact <${artifact}> .
123+
?dataset dct:hasVersion ?v .
124+
?dataset dcat:distribution ?distribution .
125+
?distribution databus:file ?file .
126+
}
127+
}
128+
ORDER BY DESC (STR(?v)) LIMIT 1
129+
"""
130+
}
131+
132+
133+
pipeline {
134+
agent any
135+
stages {
136+
137+
stage("latest artifact file"){
138+
steps{
139+
script{
140+
def body = req(
141+
"https://databus.dbpedia.org/kikiriki/jenkins/jenkins"
142+
)
143+
// wrap in a json (x-www-urlencoded also works)
144+
def jsonBody = new groovy.json.JsonBuilder(query: body).toPrettyString()
145+
echo "Query is: \n${body}"
146+
147+
// send post http-request to a databus SPARQL endpoint
148+
def response = httpRequest validResponseCodes: "200",
149+
consoleLogResponseBody: true,
150+
httpMode: 'POST', quiet: true,
151+
requestBody: jsonBody,
152+
url: "https://databus.dbpedia.org/sparql",
153+
customHeaders:[
154+
[name: "Content-Type", value: "application/json"],
155+
[name: "Accept", value: "text/csv"]
156+
]
157+
// if we configure Accept: text/csv the endpoint returns this:
158+
// "file"
159+
// "https://databus.dbpedia.org/kikiriki/jenkins/jenkins/2024-04-09-9/jenkins.txt"
160+
echo "Response: ${response.content}"
161+
// we extract the URI from the response
162+
def fn = response.content.split('\n')[1].replaceAll('"', '').trim()
163+
164+
echo "Download URI: ${fn}"
165+
// we can use the URI to download the file using curl
166+
sh "curl -O ${fn}"
167+
}
168+
}
169+
}
170+
}
171+
172+
}
173+
```

‎server/app/api/swagger.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -110,8 +110,8 @@ paths:
110110
"@context": "https://downloads.dbpedia.org/databus/context.jsonld",
111111
"@graph": [
112112
{
113-
"@type": "Dataset",
114-
"@id": "https://dev.databus.dbpedia.org/<your username>/test_group/test_artifact/2022-02-09#Dataset",
113+
"@type": "Version",
114+
"@id": "https://dev.databus.dbpedia.org/<your username>/test_group/test_artifact/2022-02-09",
115115
"hasVersion": "2022-02-09",
116116
"title": "test_group test_artifact",
117117
"description": "version of the test_artifact dataset from DBpedia",

0 commit comments

Comments
 (0)