[SPARK-56500][R] Replace devtools to remotes in SparkR

Replaces `devtools` to `remotes` which is lighter.
It does not fix CRAN related because we don't release on CRAN anymore.

- A lot of usages of `devtools` in our repo is actually deprecated.
- Causes many issues like #55216 #55233 #55245
- It fails again because it requires another dependency.

No, dev-only.

Will minotior the CI.

No.

Closes #55362 from HyukjinKwon/SPARK-56500.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 84e6954)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

Author: HyukjinKwon

.github/workflows/build_and_test.yml

@@ -689,8 +689,8 @@ jobs:
         apt-get install -y libcurl4-openssl-dev libgit2-dev libssl-dev libxml2-dev \
           libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev \
           libtiff5-dev libjpeg-dev
-        Rscript -e "install.packages(c('devtools'), repos='https://cloud.r-project.org/')"
-        Rscript -e "devtools::install_version('lintr', version='2.0.1', repos='https://cloud.r-project.org')"
+        Rscript -e "install.packages(c('remotes'), repos='https://cloud.r-project.org/')"
+        Rscript -e "remotes::install_version('lintr', version='2.0.1', repos='https://cloud.r-project.org')"
         ./R/install-dev.sh
     - name: Install dependencies for documentation generation
       run: |
@@ -708,9 +708,9 @@ jobs:
         python3.9 -m pip install 'docutils<0.18.0' # See SPARK-39421
         apt-get update -y
         apt-get install -y ruby ruby-dev
-        Rscript -e "install.packages(c('devtools', 'testthat', 'knitr', 'rmarkdown', 'markdown', 'e1071', 'roxygen2', 'ggplot2', 'mvtnorm', 'statmod'), repos='https://cloud.r-project.org/')"
-        Rscript -e "devtools::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
-        Rscript -e "devtools::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')"
+        Rscript -e "install.packages(c('remotes', 'testthat', 'knitr', 'rmarkdown', 'markdown', 'e1071', 'roxygen2', 'ggplot2', 'mvtnorm', 'statmod'), repos='https://cloud.r-project.org/')"
+        Rscript -e "remotes::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
+        Rscript -e "remotes::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')"
         gem install bundler -v 2.4.22
         cd docs
         bundle install

R/DOCUMENTATION.md

@@ -22,8 +22,7 @@ SparkR documentation is generated by using in-source comments and annotated by u
 [`roxygen2`](https://cran.r-project.org/package=roxygen2). After making changes to the documentation and generating man pages,
 you can run the following from an R console in the SparkR home directory
 ```R
-library(devtools)
-devtools::document(pkg="./pkg", roclets=c("rd"))
+roxygen2::roxygenize(package.dir="./pkg", roclets=c("rd"))
 ```
 You can verify if your changes are good by running
 

R/create-docs.sh

@@ -18,7 +18,7 @@
 #
 
 # Script to create API docs and vignettes for SparkR
-# This requires `devtools`, `knitr` and `rmarkdown` to be installed on the machine.
+# This requires `roxygen2`, `knitr`, `rmarkdown`, and `pkgdown` to be installed on the machine.
 
 # After running this script the html docs can be found in
 # $SPARK_HOME/R/pkg/html

R/create-rd.sh

@@ -33,5 +33,5 @@ FWDIR="$(cd "`dirname "${BASH_SOURCE[0]}"`"; pwd)"
 pushd "$FWDIR" > /dev/null
 . "$FWDIR/find-r.sh"
 
-# Generate Rd files if devtools is installed
-"$R_SCRIPT_PATH/Rscript" -e ' if(requireNamespace("devtools", quietly=TRUE)) { setwd("'$FWDIR'"); devtools::document(pkg="./pkg", roclets="rd") }'
+# Generate Rd files if roxygen2 is installed
+"$R_SCRIPT_PATH/Rscript" -e ' if(requireNamespace("roxygen2", quietly=TRUE)) { setwd("'$FWDIR'"); roxygen2::roxygenize(package.dir="./pkg", roclets=c("rd")) }'

R/pkg/README.md

@@ -53,7 +53,7 @@ Once you have made your changes, please include unit tests for them and run exis
 
 #### Generating documentation
 
-The SparkR documentation (Rd files and HTML files) are not a part of the source repository. To generate them you can run the script `R/create-docs.sh`. This script uses `devtools` and `knitr` to generate the docs and these packages need to be installed on the machine before using the script. Also, you may need to install these [prerequisites](https://github.com/apache/spark/tree/master/docs#prerequisites). See also, `R/DOCUMENTATION.md`
+The SparkR documentation (Rd files and HTML files) are not a part of the source repository. To generate them you can run the script `R/create-docs.sh`. This script uses `roxygen2`, `knitr`, and `rmarkdown` to generate the docs and these packages need to be installed on the machine before using the script. Also, you may need to install these [prerequisites](https://github.com/apache/spark/tree/master/docs#prerequisites). See also, `R/DOCUMENTATION.md`
 
 ### Examples, Unit tests
 

dev/ansible-for-test-node/roles/jenkins-worker/tasks/install_spark_build_packages.yml

@@ -165,7 +165,7 @@
   loop:
     - digest
     - knitr
-    - devtools
+    - remotes
     - plyr
     - roxygen2
     - rmarkdown
@@ -179,5 +179,5 @@
   register: r_check
 
 - name: install lintr v2.0.0
-  command: /usr/bin/Rscript --no-echo --no-save --no-restore-history -e "devtools::install_github('jimhester/lintr@v2.0.0')"
+  command: /usr/bin/Rscript --no-echo --no-save --no-restore-history -e "remotes::install_github('jimhester/lintr@v2.0.0')"
   when: "'lintr' not in r_check.stdout"

dev/create-release/spark-rm/Dockerfile.base

@@ -91,13 +91,13 @@ RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/' >> /et
 
 # Install R packages (same versions across all branches)
 # See more in SPARK-39959, roxygen2 < 7.2.1
-RUN Rscript -e "install.packages(c('devtools', 'remotes', 'knitr', 'markdown', \
+RUN Rscript -e "install.packages(c('remotes', 'knitr', 'markdown', \
     'rmarkdown', 'testthat', 'e1071', 'survival', 'arrow', \
     'ggplot2', 'mvtnorm', 'statmod', 'xml2'), repos='https://cloud.r-project.org/')" && \
-    Rscript -e "devtools::install_version('roxygen2', version='7.2.0', repos='https://cloud.r-project.org')" && \
-    Rscript -e "devtools::install_version('lintr', version='2.0.1', repos='https://cloud.r-project.org')" && \
-    Rscript -e "devtools::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')" && \
-    Rscript -e "devtools::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
+    Rscript -e "remotes::install_version('roxygen2', version='7.2.0', repos='https://cloud.r-project.org')" && \
+    Rscript -e "remotes::install_version('lintr', version='2.0.1', repos='https://cloud.r-project.org')" && \
+    Rscript -e "remotes::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')" && \
+    Rscript -e "remotes::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
 
 # See more in SPARK-39735
 ENV R_LIBS_SITE="/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"

dev/infra/Dockerfile

@@ -52,14 +52,14 @@ RUN gpg -a --export E084DAB9 | apt-key add -
 RUN add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/'
 RUN apt update
 RUN $APT_INSTALL r-base libcurl4-openssl-dev qpdf libssl-dev zlib1g-dev
-RUN Rscript -e "install.packages(c('knitr', 'markdown', 'rmarkdown', 'testthat', 'devtools', 'e1071', 'survival', 'arrow', 'roxygen2', 'xml2'), repos='https://cloud.r-project.org/')"
+RUN Rscript -e "install.packages(c('remotes', 'knitr', 'markdown', 'rmarkdown', 'testthat', 'e1071', 'survival', 'arrow', 'roxygen2', 'xml2'), repos='https://cloud.r-project.org/')"
 
 # See more in SPARK-39959, roxygen2 < 7.2.1
 RUN apt-get install -y libcurl4-openssl-dev libgit2-dev libssl-dev libxml2-dev \
           libfontconfig1-dev libharfbuzz-dev libfribidi-dev libfreetype6-dev libpng-dev \
           libtiff5-dev libjpeg-dev
-RUN Rscript -e "install.packages(c('devtools'), repos='https://cloud.r-project.org/')"
-RUN Rscript -e "devtools::install_version('roxygen2', version='7.2.0', repos='https://cloud.r-project.org')"
+RUN Rscript -e "install.packages(c('remotes'), repos='https://cloud.r-project.org/')"
+RUN Rscript -e "remotes::install_version('roxygen2', version='7.2.0', repos='https://cloud.r-project.org')"
 
 # See more in SPARK-39735
 ENV R_LIBS_SITE "/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"

docs/README.md

@@ -70,10 +70,10 @@ If you'd like to generate R API documentation, you'll need to [install Pandoc](h
 and install these libraries:
 
 ```sh
-$ sudo Rscript -e 'install.packages(c("knitr", "devtools", "testthat", "rmarkdown"), repos="https://cloud.r-project.org/")'
-$ sudo Rscript -e 'devtools::install_version("roxygen2", version = "7.1.2", repos="https://cloud.r-project.org/")'
-$ sudo Rscript -e "devtools::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
-$ sudo Rscript -e "devtools::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')"
+$ sudo Rscript -e 'install.packages(c("remotes", "knitr", "testthat", "rmarkdown"), repos="https://cloud.r-project.org/")'
+$ sudo Rscript -e 'remotes::install_version("roxygen2", version = "7.1.2", repos="https://cloud.r-project.org/")'
+$ sudo Rscript -e "remotes::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
+$ sudo Rscript -e "remotes::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')"
 ```
 
 Note: Other versions of roxygen2 might work in SparkR documentation generation but `RoxygenNote` field in `$SPARK_HOME/R/pkg/DESCRIPTION` is 7.1.2, which is updated if the version is mismatched.

docs/building-spark.md

@@ -241,7 +241,7 @@ The run-tests script also can be limited to a specific Python version or a speci
 
 To run the SparkR tests you will need to install the [knitr](https://cran.r-project.org/package=knitr), [rmarkdown](https://cran.r-project.org/package=rmarkdown), [testthat](https://cran.r-project.org/package=testthat), [e1071](https://cran.r-project.org/package=e1071) and [survival](https://cran.r-project.org/package=survival) packages first:
 
-    Rscript -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')"
+    Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')"
 
 You can run just the SparkR tests using the command: