Getting Started

About INSPIRE

About

Inspire is a set of services, the main one being a search engine for high energy physics papers, with some side services like authors profiles, conferences, journals, institutions, experiments and a small specialized job market. It’s main purpose is to provide physicists worldwide with a source of information about high energy physics related topics.

Currently we have two main websites open to the public:

  • The Legacy website, with the current production application.
  • The QA website, running the latest inspire-next code (for test purposes only).

Installing INSPIRE

Docker (Linux)

Docker is an application that makes it simple and easy to run processes in a container, which are like virtual machines, but more resource-friendly. For a detailed introduction to the different components of a Docker container, you can follow this tutorial.

Inspire and Docker

Get the latest Docker appropriate to your operationg system, by visiting Docker’s official web site and accessing the Get Docker section.

Note

If you are using Mac, please build a simple box with docker-engine above 1.10 and docker-compose above 1.6.0.

Make sure you can run docker without sudo.

  • id $USER

    If you are not in the docker group, run the following command and then restart docker. If this doesn’t work, just restart your machine :)

  • newgrp docker or su - $USER

  • sudo usermod -a -G docker $USER

Get the latest docker-compose:

$ sudo pip install docker-compose
  • Add DOCKER_DATA env variable in your .bashrc or .zshrc. In this directory you will have all the persistent data between Docker runs.
$ export DOCKER_DATA=~/inspirehep_docker_data/
$ mkdir -p "$DOCKER_DATA"

By default the virtualenv and everything else will be kept on /tmp and they will be available only until the next reboot.

  • Install a host persistent venv and build assets

Note

From now on all the docker-compose commands must be run at the root of the inspire-next repository, you can get a local copy with:

$ git clone git://github.com/inspirehep/inspire-next
$ cd inspire-next

$ docker-compose pull
$ docker-compose -f docker-compose.deps.yml run --rm pip

Note

If you have trouble with internet connection inside docker probably you are facing known DNS issue. Please follow this solution with DNS: --dns 137.138.17.5 --dns 137.138.16.5.

$ docker-compose -f docker-compose.deps.yml run --rm assets
  • Run the service locally
$ docker-compose up
  • Populate database
$ docker-compose run --rm web scripts/recreate_records

Once you have the database populated with the tables and demo records, you can go to localhost:5000

  • Run tests in an isolated environment.

Note

The tests use a different set of containers than the default docker-compose up, so if you run both at the same time you might start having ram/load issues, if so, you can stop all the containers started by docker-compose up with docker-compose kill -f

You can choose one of the following tests types:

  • unit
  • workflows
  • integration
  • acceptance-authors
  • acceptance-literature
$ docker-compose -f docker-compose.test.yml run --rm <tests type>
$ docker-compose -f docker-compose.test.yml down

Tip

  • cleanup all the containers:

    docker rm $(docker ps -qa)

  • cleanup all the images:

    docker rmi $(docker images -q)

  • cleanup the virtualenv (careful, if docker_data is set to something you care about, it will be removed):

    sudo rm -rf "${DOCKER_DATA?DOCKER_DATA was not set, ignoring}"

Extra useful tips

  • Run a random shell
$ docker-compose run --rm web inspirehep shell

$ docker-compose run --rm web bash
  • Reload code in a worker
$ docker-compose restart worker
  • Quick and safe reindex
$ docker-compose restart worker && docker-compose run --rm web scripts/recreate_records
  • Recreate all static assets. Will download all dependencies from npm and copy all static files to ${DOCKER_DATA}/tmp/virtualenv/var/inspirehep-instance/static.
$ docker-compose -f docker-compose.deps.yml run --rm assets
  • Monitor the output from all the services (elasticsearch, web, celery workers, database, flower, rabbitmq, scrapyd, redis) via the following command:
$ docker-compose up

Native Install (CentOS - MacOS)

System prerequisites

This guide expects you to have installed in your system the following tools:

  • git
  • virtualenv
  • virtualenvwrapper
  • npm > 3.0
  • postgresql + devel headers
  • libxml2 + devel headers
  • libxslt + devel headers
  • ImageMagick
  • redis
  • elasticsearch
CentOS
$ sudo yum install python-virtualenv python-virtualenvwrapper \
    npm postgresql postgresql-devel libxml2-devel ImageMagick redis git \
    libxslt-devel
$ sudo npm -g install npm

For elasticsearch you can find the installation instructions on the elasticsearch install page, and, to run the development environment, you will need also to add the following workarounds:

$ sudo usermod -a -G $USER elasticsearch
$ newgrp elasticsearch  # or log out and in again
$ sudo ln -s /etc/elasticsearch /usr/share/elasticsearch/config
MacOS
$ brew install postgresql
$ brew install libxml2
$ brew install libxslt
$ brew install redis
$ brew cask install caskroom/versions/java8
$ brew install elasticsearch@2.4
$ brew install rabbitmq
$ brew install imagemagick@6
$ brew install libmagic
$ brew install ghostscript
$ brew install poppler

You might also need to link imagemagick:

$ brew link --force imagemagick@6

Add to ~/.bash_profile:

# ElasticSearch.
export PATH="/usr/local/opt/elasticsearch@2.4/bin:$PATH"

Create a virtual environment

Create a virtual environment and clone the INSPIRE source code using git:

$ mkvirtualenv --python=python2.7 inspirehep
$ workon inspirehep
(inspirehep)$ cdvirtualenv
(inspirehep)$ mkdir src
(inspirehep)$ git clone https://github.com/inspirehep/inspire-next.git src/inspirehep

Note

It is also possible (and more flexible) to do the above the other way around like this and clone the project into a folder of your choice:

$ git clone https://github.com/inspirehep/inspire-next.git inspirehep
$ cd inspirehep
$ mkvirtualenv --python=python2.7 inspirehep
$ workon inspirehep

This approach enables you to switch to a new virtual environment without having to clone the project again. You simply specify on which environment you want to workon using its name.

Just be careful to replace all cdvirtualenv src/inspirehep in the following with a cd path_you_chose/inspirehep.

Install requirements

Use pip to install all requirements, it’s recommended to upgrade pip and setuptools to latest too:

(inspirehep)$ pip install --upgrade pip setuptools
(inspirehep)$ cdvirtualenv src/inspirehep
(inspirehep)$ pip install -r requirements.txt --pre --exists-action i
(inspirehep)$ pip install honcho

And for development:

(inspirehep)$ pip install -e .[development]

Custom configuration and debug mode

If you want to change the database url, or enable the debug mode for troubleshooting, you can do so in the inspirehep.cfg file under var/inspirehep-instance, you might need to create it:

(inspirehep)$ cdvirtualenv var/inspirehep-instance
(inspirehep)$ vim inspirehep.cfg

There you can change the value of any of the variables that are set under the file src/inspirehep/inspirehep/config.py, for example:

DEBUG = True
SQLALCHEMY_DATABASE_URI = "postgresql+psycopg2://someuser:somepass@my.postgres.server:5432/inspirehep"

Note

Make sure that the configuration keys you override here have the same exact name as the ones in the config.py file, as it will not complain if you put a key that did not exist.

Build assets

We build assets using npm. Make sure you have installed it system wide.

(inspirehep)$ sudo npm update
(inspirehep)$ sudo npm install -g node-sass@3.8.0 clean-css@^3.4.24 requirejs uglify-js

Note

If you don’t want to use sudo to install the npm packages globally, you can still setup a per-user npm modules installation that will allow you to install/remove modules as normal user. You can find more info in the npm docs here.

In particular, if you want to install the npm packages directly in your virtualenv, just add NPM_CONFIG_PREFIX=$VIRTUAL_ENV in the postactivate file of your virtualenv folder and you will be able to run the above command from inside your virtual environment.

Then we build the INSPIRE assets:

(inspirehep)$ inspirehep npm
(inspirehep)$ cdvirtualenv var/inspirehep-instance/static
(inspirehep)$ npm install
(inspirehep)$ inspirehep collect -v
(inspirehep)$ inspirehep assets build

Note

Alternatively, run sh scripts/clean_assets to do the above in one command.

Create database

We will use postgreSQL as database. Make sure you have installed it system wide.

Then create the database and database tables if you haven’t already done so:

(inspirehep)$ psql
# CREATE USER inspirehep WITH PASSWORD 'dbpass123';
# CREATE DATABASE inspirehep;
# GRANT ALL PRIVILEGES ON DATABASE inspirehep to inspirehep;
(inspirehep)$ inspirehep db init
(inspirehep)$ inspirehep db create

Start all services

Rabbitmq

You must have rabbitmq installed and running (and reachable) somewhere. To run it locally on a CentOS:

$ sudo yum install rabbitmq-server
$ sudo service rabbitmq-server start
$ sudo systemctl enable rabbitmq-server.service  # to start on system boot
Everything else: Honcho

We use honcho to manage our services and run the development server. See Procfile for details.

(inspirehep)$ cdvirtualenv src/inspirehep
(inspirehep)$ honcho start

In MacOS you still need to manually run rabbitmq and postgresql:

$ brew services start rabbitmq
$ brew services start postgresql

And the site is now available on http://localhost:5000.

Create ElasticSearch Indices and Aliases

Note

Remember that you’ll need to have the elasticsearch bin directory in your $PATH or prepend the binaries executed with the path to the elasticsearch bin directory in your system.

First of all, we will need to install the analysis-icu elasticsearch plugin.

(inspirehep)$ plugin install analysis-icu

For MacOS the plugin command will probably not be available system wide, so:

$ /usr/local/Cellar/elasticsearch\@2.4/2.4.6/libexec/bin/plugin install analysis-icu

Now we are ready to create the indexes:

(inspirehep)$ inspirehep index init

If you are having troubles creating your indices, e.g. due to index name changes or existing legacy indices, try:

(inspirehep)$ inspirehep index destroy --force --yes-i-know
(inspirehep)$ inspirehep index init

Create admin user

Now you can create a sample admin user, for that we will use the fixtures:

(inspirehep)$ inspirehep fixtures init

Note

If you are not running in debug mode, remember to add the local=1 HTTP GET parameter to the login url so it will show you the login form, for example:

Add demo records

(inspirehep)$ cdvirtualenv src/inspirehep
(inspirehep)$ inspirehep migrate file --force --wait inspirehep/demosite/data/demo-records.xml.gz

Note

Alternatively, run sh scripts/recreate_records to drop db/index/records and re-create them in one command, it will also create the admin user.

Warning

Remember to keep honcho running in a separate window.

Create regular user

Now you can create regular users (optional) with the command:

(inspirehep)$ inspirehep users create your@email.com -a

Access the records (web/rest)

While running honcho you can access the records at

$ firefox http://localhost:5000/literature/1
$ curl -i -H "Accept: application/json" http://localhost:5000/api/records/1