Skip to content

Ci/speedup dependency install #678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
timcadman wants to merge 1 commit into
base: v7.0-dev
Choose a base branch
from
+38 −11
Draft

Ci/speedup dependency install #678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions armadillo_azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,18 @@ jobs:
condition: succeeded()


#####################################################################################
# Restore the R package library from a previous run. Combined with the
# "install only missing packages" logic in the next step, a cache hit makes
# dependency installation near-instant. Bump the 'v1' token to rebuild the cache
# (e.g. when the package list below changes).
- task: Cache@2
inputs:
key: 'rpkgs | "$(Agent.OS)" | v1'
path: '/usr/local/lib/R/site-library'
displayName: 'Restore cached R packages'


#####################################################################################
# Install R and all the dependencies dsBaseClient requires.
# If previous steps have failed then don't run.
Expand All @@ -134,11 +146,13 @@ jobs:

sudo apt-get install -qq libxml2-dev libcurl4-openssl-dev libssl-dev libgsl-dev libgit2-dev r-base -y
sudo apt-get install -qq libharfbuzz-dev libfribidi-dev libmagick++-dev libudunits2-dev libuv1-dev -y
sudo R -q -e "install.packages(c('devtools','covr'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('fields','meta','metafor','ggplot2','gridExtra','data.table'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('DSI','DSOpal','DSLite'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('MolgenisAuth', 'MolgenisArmadillo', 'DSMolgenisArmadillo'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('DescTools','e1071'), dependencies=TRUE, repos='https://cloud.r-project.org')"

# Use Posit Public Package Manager, which serves precompiled binary packages
# for this Ubuntu release instead of source - so dependencies are downloaded
# rather than compiled (the slow part). The HTTPUserAgent option is what makes
# the manager hand back binaries. The setdiff() only installs packages not
# already present, so a restored (cached) site-library is reused.
sudo R -q -e "options(repos=c(CRAN=paste0('https://packagemanager.posit.co/cran/__linux__/', system('lsb_release -cs', intern=TRUE), '/latest')), HTTPUserAgent=sprintf('R/%s R (%s)', getRversion(), paste(getRversion(), R.version\$platform, R.version\$arch, R.version\$os)), Ncpus=4); pkgs <- c('devtools','covr','fields','meta','metafor','ggplot2','gridExtra','data.table','DSI','DSOpal','DSLite','MolgenisAuth','MolgenisArmadillo','DSMolgenisArmadillo','DescTools','e1071'); to_install <- setdiff(pkgs, rownames(installed.packages())); if (length(to_install)) install.packages(to_install, dependencies=TRUE)"

sudo R -q -e "library('devtools'); devtools::install_github(repo='datashield/dsDangerClient', ref='v6.3.4-dev', dependencies = TRUE)"

Expand Down
25 changes: 19 additions & 6 deletions opal_azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,18 @@ jobs:
condition: succeeded()


#####################################################################################
# Restore the R package library from a previous run. Combined with the
# "install only missing packages" logic in the next step, a cache hit makes
# dependency installation near-instant. Bump the 'v1' token to rebuild the cache
# (e.g. when the package list below changes).
- task: Cache@2
inputs:
key: 'rpkgs | "$(Agent.OS)" | opal | v1'
path: '/usr/local/lib/R/site-library'
displayName: 'Restore cached R packages'


#####################################################################################
# Install R and all the dependencies dsBaseClient requires.
# If previous steps have failed then don't run.
Expand All @@ -115,12 +127,13 @@ jobs:

sudo apt-get install -qq libxml2-dev libcurl4-openssl-dev libssl-dev libgsl-dev libgit2-dev r-base -y
sudo apt-get install -qq libharfbuzz-dev libfribidi-dev libmagick++-dev libudunits2-dev libuv1-dev -y
sudo R -q -e "install.packages(c('curl','httr'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('devtools','covr'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('fields','meta','metafor','ggplot2','gridExtra','data.table'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('DSI','DSOpal','DSLite'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('MolgenisAuth', 'MolgenisArmadillo', 'DSMolgenisArmadillo'), dependencies=TRUE, repos='https://cloud.r-project.org')"
sudo R -q -e "install.packages(c('DescTools','e1071'), dependencies=TRUE, repos='https://cloud.r-project.org')"

# Use Posit Public Package Manager, which serves precompiled binary packages
# for this Ubuntu release instead of source - so dependencies are downloaded
# rather than compiled (the slow part). The HTTPUserAgent option is what makes
# the manager hand back binaries. The setdiff() only installs packages not
# already present, so a restored (cached) site-library is reused.
sudo R -q -e "options(repos=c(CRAN=paste0('https://packagemanager.posit.co/cran/__linux__/', system('lsb_release -cs', intern=TRUE), '/latest')), HTTPUserAgent=sprintf('R/%s R (%s)', getRversion(), paste(getRversion(), R.version\$platform, R.version\$arch, R.version\$os)), Ncpus=4); pkgs <- c('curl','httr','devtools','covr','fields','meta','metafor','ggplot2','gridExtra','data.table','DSI','DSOpal','DSLite','MolgenisAuth','MolgenisArmadillo','DSMolgenisArmadillo','DescTools','e1071'); to_install <- setdiff(pkgs, rownames(installed.packages())); if (length(to_install)) install.packages(to_install, dependencies=TRUE)"

sudo R -q -e "library('devtools'); devtools::install_github(repo='datashield/dsDangerClient', ref='6.3.4', dependencies = TRUE)"

Expand Down