Common Tasks

Caching

For caching we use Invenio-Cache. For example, to set a value in the cache:

>>> from invenio_cache import current_cache
>>> current_cache.set('test', [1, 2, 3], timeout=60)

And to retrieve the value from the cache:

>>> from invenio_cache import current_cache
>>> current_cache.get('test')

Profiling a Celery Task

To profile a Celery task we need to make sure that the task is executed by the same Python process in which we are collecting the profiling information. That is, the configuration must contain

CELERY_TASK_ALWAYS_EAGER = True
CELERY_RESULT_BACKEND = 'cache'
CELERY_CACHE_BACKEND = 'memory'

Then, in a Flask shell, we do

>>> import cProfile
>>> import pstats
>>> from path.to.our.task import task
>>> pr = cProfile.Profile()
>>> pr.runcall(task, *args, **kwargs)

where *args and *kwargs are the arguments and keyword arguments that we want to pass to task. Then

>>> ps = pstats.Stats(pr)
>>> ps.dump_stats('task.prof')

will create a binary file containing the desired profiling information. To read it we can use snakeviz, which will create a graph such as

An example of a snakeviz graph.

Essentially each layer of the graph is a level of the call stack, and the size of the slice is the total time of the function call. For a complete explanation visit the documentation of snakeviz.

Profiling a Request

To profile a request we need to add the following variable to our configuration:

PROFILE = True

Then we need to attach the WSGI application profiler to our WSGI application. To do this, we need to add a few lines at the bottom of inspirehep/wsgi.py:

import os; os.mkdir('prof')
from werkzeug.contrib.profiler import ProfilerMiddleware
application = ProfilerMiddleware(application, profile_dir='prof')

Now, after we restart the application, a profile report will be created in the prof folder for each request that we make. These binary files can be visualized as above with snakeviz.

Rebuild the assets (js/css bundles)

From the root of the code repository, you can run the helper script:

$ workon inspire
(inspire)$ ./scripts/clean_assets

This will:

  1. Remove all your static assets
  2. Gather all the npm dependencies and write them in the file package.json in the instance static folder
  3. Execute npm install
  4. Execute inspirehep collect and inspirehep assets build

You should then find all your updated assets in the static folder of your inspire installation, if you are using virtualenv:

cdvirtualenv var/inspirehep-instance/static/

Rebuild the database, the elasticsearch indexes, and reupload the demo records

Same as the assets, from the root of the code repository, run the script:

$ workon inspire
(inspire)$ ./scripts/recreate_records