Skip to content

Commit fdcd077

Browse files
Marian Zvadadjw8605
Marian Zvada
authored and
djw8605
committed
Docs for StashCache (#62)
Adding full documentation on installing and configuring StashCache cache and origin servers. This documentation should be helpful for both system admins that want to install a Cache or Origin server, as well as StashCache Operators.
1 parent e2e242e commit fdcd077

23 files changed

Lines changed: 722 additions & 37 deletions

‎docs/admin/configure-cache-auth.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Configuring Cache Server with authentication
2+
3+
Before you continue, make sure [default Cache Server](configure-cache.md) is configured on first place. Enabling authenticated cache is optionali and additional do default cache instance. This chapter describes all the steps needed.
4+
5+
Packages installed: `stashcache-daemon fetch-crl stashcache-cache-server xrootd-lcmaps globus-proxy-utils`
6+
7+
## Authenticated Cache server
8+
9+
:heavy_exclamation_mark: Make sure you've in place following pre-requisities from [install step here](install.md):
10+
* __Service certificate:__ create copy of the certificate to `/etc/grid-security/xrd/xrd{cert,key}.pem`
11+
* set owner of the directory `/etc/grid-security/xrd/` to `xrootd:xrootd` user:
12+
```
13+
$ chown -R xrootd:xrootd /etc/grid-security/xrd/
14+
```
15+
* __Network ports__: allow connections on port `8443 (TCP)`
16+
17+
:heavy_exclamation_mark: Beware, authenticated cache requires presence of the [config file](../configs/xrootd-stashcache-cache-server.cfg) `/etc/xrootd/xrootd-stashcache-cache-server.cfg`.
18+
19+
Now, create symbolic link to existing configuration file with `-auth` postfix:
20+
```
21+
[root@client ~]$ cd /etc/xrootd/
22+
[root@client ~]$ ln -s xrootd-stashcache-cache-server.cfg xrootd-stashcache-cache-server-auth.cfg
23+
```
24+
25+
### RHEL7
26+
27+
On RHEL7 system, you need to configure and run following systemd units:
28+
* `xrootd@stashcache-cache-server-auth.service`
29+
* `xrootd-renew-proxy.service`
30+
* `xrootd-renew-proxy.timer`
31+
* `fetch-crl-cron`
32+
33+
#### Auth.service
34+
1. Enable `xrootd@stashcache-cache-server-auth.service` instance:
35+
```
36+
[root@client ~]$ systemctl enable xrootd@stashcache-cache-server-auth
37+
```
38+
39+
2. Reload daemons:
40+
```
41+
[root@client ~]$ systemctl daemon-reload
42+
```
43+
44+
#### Proxy.service
45+
1. Create the file with following content:
46+
```
47+
[root@client ~]$ cat /usr/lib/systemd/system/xrootd-renew-proxy.service
48+
[Unit]
49+
Description=Renew xrootd proxy
50+
51+
[Service]
52+
User=xrootd
53+
Group=xrootd
54+
Type = oneshot
55+
ExecStart = /bin/grid-proxy-init -cert /etc/grid-security/xrd/xrdcert.pem -key /etc/grid-security/xrd/xrdkey.pem -out /tmp/x509up_xrootd -valid 48:00
56+
57+
[Install]
58+
WantedBy=multi-user.target
59+
```
60+
61+
2. Reload daemons:
62+
```
63+
[root@client ~]$ systemctl daemon-reload
64+
```
65+
66+
#### Proxy.timer
67+
1. Create the file with following content:
68+
```
69+
[root@client ~]$ cat /usr/lib/systemd/system/xrootd-renew-proxy.timer
70+
[Unit]
71+
Description=Renew proxy every day at midnight
72+
73+
[Timer]
74+
OnCalendar=*-*-* 00:00:00
75+
Unit=xrootd-renew-proxy.service
76+
77+
[Install]
78+
WantedBy=multi-user.target
79+
```
80+
81+
2. Enable timer:
82+
```
83+
[root@client ~]$ systemctl enable xrootd-renew-proxy.timer
84+
```
85+
86+
3. Start and check if timer is active and working:
87+
```
88+
[root@client ~]$ systemctl start xrootd-renew-proxy.timer
89+
...
90+
[root@client ~]$ systemctl is-active xrootd-renew-proxy.timer
91+
active
92+
[root@client ~]$ systemctl list-timers xrootd-renew-proxy*
93+
NEXT LEFT LAST PASSED UNIT ACTIVATES
94+
Thu 2017-05-11 00:00:00 CDT 54min left Wed 2017-05-10 00:00:01 CDT 23h ago xrootd-renew-proxy.timer xrootd-renew-proxy.service
95+
```
96+
97+
4. Reload daemons:
98+
```
99+
[root@client ~]$ systemctl daemon-reload
100+
```
101+
102+
#### CRLs updates
103+
It is very important to keep CRL list updated from cron:
104+
1. Enable fetch-crl-cron
105+
```
106+
[root@client ~]$ systemctl enable fetch-crl-cron
107+
```
108+
109+
2. Start fetch-crl-cron
110+
```
111+
[root@client ~]$ systemctl start fetch-crl-cron
112+
```
113+
114+
3. Reload daemons:
115+
```
116+
[root@client ~]$ systemctl daemon-reload
117+
```
118+
119+
### RHEL6
120+
...to be added
121+
122+
### Add Authfile for authenticated cache
123+
Authfile for authenticated cache may differ from `/etc/xrootd/Authfile-noauth` defined in [non-authenticated cache configuration](configure-cache.md). Example:
124+
```
125+
[root@client ~]$ cat /etc/xrootd/Authfile-auth
126+
g /osg/ligo /user/ligo r
127+
u ligo /user/ligo lr / rl
128+
```
129+
130+
When ready with configuration, please [register](../ops/register.md) and [start](../ops/start.md) your StashCache Cache server.

‎docs/admin/configure-cache-optional.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Optional configuration
2+
3+
## Adjust disk utilization
4+
To adjust the disk utilization of your StashCache cache, modify the values of `pfc.diskusage` in `/etc/xrootd/xrootd-stashcache-cache-server.cfg`:
5+
```
6+
pfc.diskusage 0.98 .99
7+
```
8+
The first value and second values correspond to the low and high usage watermarks, respectively, in percentages. When the high watermark is reached, the XRootD service will automatically purge cache objects down to the low watermark.
9+
10+
## Enable remote debugging
11+
This feature enables remote debugging via the `digFS` read-only file system, it's optional line in the [config file](../configs/xrootd-stashcache-cache-server.cfg):
12+
```
13+
xrootd.diglib * /etc/xrootd/digauth.cf
14+
```
15+
where `/etc/xrootd/digauth.cf` may have following content:
16+
```
17+
all allow host h=abc.org
18+
all allow host h=*.xyz.edu
19+
```
20+
21+
When ready with configuration, please [register](../ops/register.md) and [start](../ops/start.md) your StashCache Cache server.

‎docs/admin/configure-cache.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Configuring Cache Server
2+
3+
Packages installed: `stashcache-daemon fetch-crl stashcache-cache-server`
4+
5+
The following section describes required configuration to have a functional non-authenticated StashCache Cache (not origin server!). StashCache Cache package `stashcache-cache-server` needs to be manually configured from pre-existing XRootD configuration.
6+
7+
## Cache server
8+
!!! Note:
9+
:bangbang: While example of the configuration file below provides combination of _authenticated_ and _non-authenticated_ _Cache_, the non-authenticated cache is considered to be default and authenticated cache just optional (additional) service. If you're about to configure in addition _authenticated cache_ read to the end of this document and then follow post-installation of [authenticated part here](configure-cache-auth.md).
10+
11+
For configuring **cache** one needs to define directive `pss.origin redirector.osgstorage.org:1024` (not `all.manager redirector.osgstorage.org+ 1213` directive as it is in case of [configuring origin](configure-origin.md)).
12+
`StashCache-daemon` package provides default configuration file `/etc/xrootd/xrootd-stashcache-cache-server.cfg`. Example of the configuration of cache server is as follows:
13+
```
14+
all.export /
15+
set cachedir = /stash
16+
xrd.allow host *
17+
sec.protocol host
18+
all.adminpath /var/spool/xrootd
19+
20+
xrootd.trace emsg login stall redirect
21+
ofs.trace all
22+
xrd.trace all
23+
cms.trace all
24+
25+
ofs.osslib libXrdPss.so
26+
pss.origin redirector.osgstorage.org:1094
27+
pss.cachelib libXrdFileCache.so
28+
pss.setopt DebugLevel 1
29+
30+
oss.localroot $(cachedir)
31+
32+
# Config for v1 (xrootd <=v4.5.0)
33+
#pfc.nramprefetch 4
34+
#pfc.nramread 4
35+
#pfc.diskusage 0.98 0.99
36+
37+
# Config for v2 (xrootd >v4.5.0)
38+
pfc.blocksize 512k
39+
pfc.ram 32g
40+
pfc.prefetch 10
41+
pfc.diskusage 0.98 0.99
42+
43+
xrootd.seclib /usr/lib64/libXrdSec.so
44+
sec.protocol /usr/lib64 gsi \
45+
-certdir:/etc/grid-security/certificates \
46+
-cert:/etc/grid-security/xrd/xrdcert.pem \
47+
-key:/etc/grid-security/xrd/xrdkey.pem \
48+
-crl:1 \
49+
-authzfun:libXrdLcmaps.so \
50+
-authzfunparms:--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,4|useglobals \
51+
-gmapopt:10 \
52+
-authzto:3600
53+
54+
# Enable the authorization module, even if we have an unauthenticated instance.
55+
ofs.authorize 1
56+
acc.audit deny grant
57+
58+
# Run the authenticated instance on port 8443 (Xrootd and HTTPS)
59+
# Notice authenticated and unauthenticated instances use separate auth
60+
# files.
61+
if named stashcache-cache-server-auth
62+
#pss.origin red-gridftp4.unl.edu:1094
63+
xrd.port 8443
64+
acc.authdb /etc/xrootd/Authfile-auth
65+
sec.protbind * gsi
66+
xrd.protocol http:8443 libXrdHttp.so
67+
pss.origin xrootd-local.unl.edu:1094
68+
else
69+
# Unauthenticated instance runs on port 1094 (Xrootd) and 8000 (HTTP/HTTPS)
70+
acc.authdb /etc/xrootd/Authfile-noauth
71+
#sec.protbind * none
72+
sec.protbind * none
73+
xrd.protocol http:8000 libXrdHttp.so
74+
fi
75+
76+
http.cadir /etc/grid-security/certificates
77+
http.cert /etc/grid-security/xrd/xrdcert.pem
78+
http.key /etc/grid-security/xrd/xrdkey.pem
79+
http.secxtractor /usr/lib64/libXrdLcmaps.so
80+
http.listingdeny yes
81+
http.staticpreload http://static/robots.txt /etc/xrootd/stashcache-robots.txt
82+
83+
# Tune the client timeouts to more aggressively timeout.
84+
pss.setopt ParallelEvtLoop 10
85+
pss.setopt RequestTimeout 25
86+
pss.setopt ConnectTimeout 25
87+
pss.setopt ConnectionRetry 2
88+
89+
#Sending monitoring information
90+
xrd.report uct2-collectd.mwt2.org:9931
91+
xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest redir fstat info user uct2-collectd.mwt2.org:9930
92+
93+
all.sitename Nebraska
94+
95+
# Optional configuration
96+
# Remote debugging
97+
xrootd.diglib * /etc/xrootd/digauth.cf
98+
```
99+
100+
### Add Authfile for non-authenticated cache
101+
In Authfile you want to allow local reads below `$(cachedir)` defined in the main config. Example of Authfile:
102+
```
103+
[root@client ~]$ cat /etc/xrootd/Authfile-noauth
104+
u * /user/ligo -rl / rl
105+
```
106+
107+
### Add Robots file
108+
```
109+
[root@client ~]$ cat /etc/xrootd/stashcache-robots.txt
110+
User-agent: *
111+
Disallow: /
112+
```
113+
114+
### RHEL7
115+
On RHEL7 system, you need to run following systemd unit:
116+
* `systemctl start xrootd@stashcache-cache-server.service`
117+
* `systemctl start condor.service`
118+
119+
Please, refer to [start services document](../ops/start.md) for more information.
120+
121+
### RHEL6
122+
...to be added, old doc for RHEL6 configuration resides temporarily [here](https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallStashCache).
123+
124+
When ready with configuration, please [register](../ops/register.md) and [start](../ops/start.md) your StashCache Cache server.
125+
If you'd like to configure in addition authenticated cache instance, please follow [this](configure-cache-auth.md) document.

‎docs/admin/configure-origin.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Configuring Origin Server
2+
3+
Packages installed: `stashcache-daemon fetch-crl stashcache-origin-server`
4+
5+
The following section describes required configuration to have a functional StashCache Origin (not cache server!). StashCache Origin package `stashcache-cache-origin` needs to be manually configured from pre-existing XRootD configuration.
6+
7+
## Origin server
8+
The origin server connects only to a redirector (not directly to cache server), thus minimal xrootd configuration is required. `StashCache-daemon` package provides default configuration file `/etc/xrootd/xrootd-stashcache-origin-server.cfg`. Example of the configuration of origin server is as follows:
9+
```
10+
all.export /
11+
set localroot = /stash
12+
xrd.port 1094
13+
14+
all.role server
15+
all.manager redirector.osgstorage.org+ 1213
16+
17+
oss.localroot $(localroot)
18+
xrootd.trace emsg login stall redirect
19+
ofs.trace none
20+
xrd.trace conn
21+
cms.trace all
22+
sec.protocol host
23+
sec.protbind * none
24+
all.adminpath /var/spool/xrootd
25+
all.pidpath /var/run/xrootd
26+
27+
# Sending monitoring information
28+
xrd.report uct2-collectd.mwt2.org:9931
29+
xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest redir fstat info user uct2-collectd.mwt2.org:9930
30+
```
31+
32+
### RHEL7
33+
On RHEL7 system, you need to run following systemd units:
34+
* `xrootd@stashcache-cache-origin.service`
35+
* `cmsd@stashcache-cache-origin.service`
36+
37+
### RHEL6
38+
...to be added
39+
40+
When ready with configuration, please [register](../ops/register.md) and [start](../ops/start.md) your StashCache Origin server.

0 commit comments

Comments
 (0)