Public PIC Wiki - User contributions [en]

AC UserManual

2018-03-19T12:57:10Z

Tonello: /* Home directory */

NOTE: '''Brackets {} in the following notes have to be removed'''. They are used to define variables.

For example: this is the home directory path for the user ''jsmith'':

/nfs/pic.es/user/{initial_user}/{user} ---> /nfs/pic.es/user/j/jsmith

== Home directory ==
Once you have your PIC account you are able to access the UI's machines:

ssh {user}@ui.pic.es

and you are you are logged in to your "home":

/nfs/pic.es/user/{initial_user}/{user}

This directory is basically thought for storing your software.
It is backed-up and has 10GiB of capacity.

== Storage ==
=== Massive storage ===
Each project has a massive storage space accessible at the following path:
/pnfs/pic.es/data/astro/{project} (see projects_url for more details)
which has only read permissions for the project's users.

Inside the directory there are two different paths corresponding to two different back-ends:

==== Tape ====
/pnfs/pic.es/data/astro/{project}/tape
As its name suggests, the data in the tape path is stored in magnetic tapes, and is critical, such as raw data or very difficult data to obtain or to get.
The size of each file is usually large, from 1-2GB to 100-200GB, due to technical reasons (they are usually ''iso'' or ''tar.bz2'' files).
Data in tapes is not very often accessed. It is necessary a previous step when accessing data in tapes, which is the pre-stage. This step is the action of reading the magnetic tape and put the data into a buffer. Once the data is in the buffer one can access the data as it was in a disk. After some time, the data is removed from the buffer. In case the data in tape has to be available for long time get in touch with the contant person of the project.

==== Disk ====
/pnfs/pic.es/data/astro/{project}/disk
Disk data is usually the data being currently used by the project, and it is being very often accessed. The size of the files is not important here.

=== Scratch ===

Each user has a scratch space at the following path:
/nfs/astro/{user}
This space is thought as a sandbox.
When the project decides some data from any scratch space is important enough for the whole collaboration, the contact person for each project will move the data into the /pnfs storage.

Take note that all '''data older than 6 months may be erased at any time''' without prior notice.

'''Any location not included in the former paths is not allowed and its contents erased on sight.'''

== Working with Python environments (any machine except Big Data platform) ==

1.1 Usually environments are all saved in the same directory (e.g. ~/env). In case it is not created:

mkdir ~/env/

1.2 Create a new environment:
cd ~/env/
virtualenv {env_name}

1.3 Activate environment:

. ~/env/{env_name}/bin/activate

1.4 Update pip command (only for the first time):

pip install -U pip

1.5 Install any package you need (in case you have any problem with some package, please contact us)

e.g. jupyter notebooks package:
pip install jupyter[notebook]

e.g numpy package:
pip install numpy

== Download code and git rules ==

These are the git rules for developers at PIC.

The methodology written below is a try to help the code development of the team and they are thought for non-experts git users.

It has been compiled from the official git documentation, which we strongly recommend to look at (at least the first three chapters):

[https://git-scm.com/book/en/v2 Git documentation]

And from this git branch model:

[http://nvie.com/posts/a-successful-git-branching-model A successful Git branching model]

We assume you already have a PIC account and you have already created a python virtual environment

Codes are hosted at https://gitlab.pic.es.

1. Download the code (first you need to have permissions to do it)

1.1. Access ui:

ssh {user}@ui.pic.es

1.2 Usually software is stored in the same directory:

mkdir ~/src/

1.2 Create a directory in which you are going to develop your codes, e.g:

mkdir -p ~/{src}/{software_project_name}

1.3 Copy the code from the gitlab repository in the created directory:

cd ~/src/{software_project_name}
git clone https://gitlab01.pic.es/{software_project_name}/{pipeline}.git

1.4 Activate your environment

. ~/env/{env_name}/bin/activate

1.5 Locate the `setup.py` file of the project (usually in the main software directory), and deploit the code:

cd ~/src/{software_project_name}
pip install -e .

2. Create your own branch

Every project has two main protected branches: ''master'' and ''develop''. Protected means you, as a standard developer, will not have permissions to write on them. Therefore in order to develop your features in the code you need to create your own branch that will always come from the ''develop'' branch.

2.1 Enter in the project directory:

cd {pipeline}

2.2 Create the branch:

git checkout -b feature_branch_name origin/develop

3. Modify the code

3.1 Day Tip:

Everyday you sit in your computer and want to modify the code, in order not to be outdated in the changes made in the develop branch, you should do:

3.1.1 Download any modification in the code

git fetch

3.1.2 Incorporate changes in the ''develop'' branch into your feature_branch_name branch

git rebase origin/develop

(Hopefully there will be no conflicts if all developers are working in ''independent'' branches. If this is not the case and you have doubts after reading the references given in point 5 below, please call us before meshing it up!)

3.2 See the changes you have done

git status

3.3 Add changes

git add changed_files

3.4 Commit changes

git commit -m "message describing the modifications"

4. Finish the new feature

Once you finish to develop, debug and test the new feature you send us an email.

We immediately will send you back another one saying that your feature has been integrated into the develop branch.

Note that your branch will be deleted.

5. Incorporate changes and start a new feature again

In order to start with a new feature you need to incorporate the changes we just did (integrate the feature into develop) and create another new branch:

git fetch

git checkout -b anoter_feature_branch_name origin/develop

AC UserManual

2018-03-19T12:55:31Z

Tonello:

NOTE: '''Brackets {} in the following notes have to be removed'''. They are used to define variables.

For example: this is the home directory path for the user ''jsmith'':

/nfs/pic.es/user/{initial_user}/{user} ---> /nfs/pic.es/user/j/jsmith

== Home directory ==
Once you have your PIC account you are able to access the UI's machines:

ssh {user}@ui.pic.es

and you are you are logged in to your "home":

/nfs/pic.es/user/{initial_user}/{user}

This directory is basically though for storing your software.
It is backup and has 10GiB of capacity.

== Storage ==
=== Massive storage ===
Each project has a massive storage space accessible at the following path:
/pnfs/pic.es/data/astro/{project} (see projects_url for more details)
which has only read permissions for the project's users.

Inside the directory there are two different paths corresponding to two different back-ends:

==== Tape ====
/pnfs/pic.es/data/astro/{project}/tape
As its name suggests, the data in the tape path is stored in magnetic tapes, and is critical, such as raw data or very difficult data to obtain or to get.
The size of each file is usually large, from 1-2GB to 100-200GB, due to technical reasons (they are usually ''iso'' or ''tar.bz2'' files).
Data in tapes is not very often accessed. It is necessary a previous step when accessing data in tapes, which is the pre-stage. This step is the action of reading the magnetic tape and put the data into a buffer. Once the data is in the buffer one can access the data as it was in a disk. After some time, the data is removed from the buffer. In case the data in tape has to be available for long time get in touch with the contant person of the project.

==== Disk ====
/pnfs/pic.es/data/astro/{project}/disk
Disk data is usually the data being currently used by the project, and it is being very often accessed. The size of the files is not important here.

=== Scratch ===

Each user has a scratch space at the following path:
/nfs/astro/{user}
This space is thought as a sandbox.
When the project decides some data from any scratch space is important enough for the whole collaboration, the contact person for each project will move the data into the /pnfs storage.

Take note that all '''data older than 6 months may be erased at any time''' without prior notice.

'''Any location not included in the former paths is not allowed and its contents erased on sight.'''

== Working with Python environments (any machine except Big Data platform) ==

1.1 Usually environments are all saved in the same directory (e.g. ~/env). In case it is not created:

mkdir ~/env/

1.2 Create a new environment:
cd ~/env/
virtualenv {env_name}

1.3 Activate environment:

. ~/env/{env_name}/bin/activate

1.4 Update pip command (only for the first time):

pip install -U pip

1.5 Install any package you need (in case you have any problem with some package, please contact us)

e.g. jupyter notebooks package:
pip install jupyter[notebook]

e.g numpy package:
pip install numpy

== Download code and git rules ==

These are the git rules for developers at PIC.

The methodology written below is a try to help the code development of the team and they are thought for non-experts git users.

It has been compiled from the official git documentation, which we strongly recommend to look at (at least the first three chapters):

[https://git-scm.com/book/en/v2 Git documentation]

And from this git branch model:

[http://nvie.com/posts/a-successful-git-branching-model A successful Git branching model]

We assume you already have a PIC account and you have already created a python virtual environment

Codes are hosted at https://gitlab.pic.es.

1. Download the code (first you need to have permissions to do it)

1.1. Access ui:

ssh {user}@ui.pic.es

1.2 Usually software is stored in the same directory:

mkdir ~/src/

1.2 Create a directory in which you are going to develop your codes, e.g:

mkdir -p ~/{src}/{software_project_name}

1.3 Copy the code from the gitlab repository in the created directory:

cd ~/src/{software_project_name}
git clone https://gitlab01.pic.es/{software_project_name}/{pipeline}.git

1.4 Activate your environment

. ~/env/{env_name}/bin/activate

1.5 Locate the `setup.py` file of the project (usually in the main software directory), and deploit the code:

cd ~/src/{software_project_name}
pip install -e .

2. Create your own branch

Every project has two main protected branches: ''master'' and ''develop''. Protected means you, as a standard developer, will not have permissions to write on them. Therefore in order to develop your features in the code you need to create your own branch that will always come from the ''develop'' branch.

2.1 Enter in the project directory:

cd {pipeline}

2.2 Create the branch:

git checkout -b feature_branch_name origin/develop

3. Modify the code

3.1 Day Tip:

Everyday you sit in your computer and want to modify the code, in order not to be outdated in the changes made in the develop branch, you should do:

3.1.1 Download any modification in the code

git fetch

3.1.2 Incorporate changes in the ''develop'' branch into your feature_branch_name branch

git rebase origin/develop

(Hopefully there will be no conflicts if all developers are working in ''independent'' branches. If this is not the case and you have doubts after reading the references given in point 5 below, please call us before meshing it up!)

3.2 See the changes you have done

git status

3.3 Add changes

git add changed_files

3.4 Commit changes

git commit -m "message describing the modifications"

4. Finish the new feature

Once you finish to develop, debug and test the new feature you send us an email.

We immediately will send you back another one saying that your feature has been integrated into the develop branch.

Note that your branch will be deleted.

5. Incorporate changes and start a new feature again

In order to start with a new feature you need to incorporate the changes we just did (integrate the feature into develop) and create another new branch:

git fetch

git checkout -b anoter_feature_branch_name origin/develop

AC UserManual

2017-03-20T09:33:14Z

Tonello: /* Create new environment */

== Storage ==
=== Massive storage ===
Each project has a massive storage space accessible at the following path:
/pnfs/pic.es/data/astro/{project} (see projects_url for more details)
which has only read permissions for the project's users.

Inside the directory there are two different paths corresponding to two different back-ends:

==== Tape ====
/pnfs/pic.es/data/astro/{project}/tape
As its name suggests, the data in the tape path is stored in magnetic tapes, and is critical, such as raw data or very difficult data to obtain or to get.
The size of each file is usually large, from 1-2GB to 100-200GB, due to technical reasons (they are usually ''iso'' or ''tar.bz2'' files).
Data in tapes is not very often accessed. It is necessary a previous step when accessing data in tapes, which is the pre-stage. This step is the action of reading the magnetic tape and put the data into a buffer. Once the data is in the buffer one can access the data as it was in a disk. After some time, the data is removed from the buffer. In case the data in tape has to be available for long time get in touch with the contant person of the project.

==== Disk ====
/pnfs/pic.es/data/astro/{project}/disk
Disk data is usually the data being currently used by the project, and it is being very often accessed. The size of the files is not important here.

=== Scratch ===

Each user has a scratch space at the following path:
/nfs/astro/{user}
This space is thought as a sandbox.
When the project decides some data from any scratch space is important enough for the whole collaboration, the contact person for each project will move the data into the /pnfs storage.

Take note that all '''data older than 6 months may be erased at any time''' without prior notice.

=== Software ===
Each user has a software space (for editable source code) at the following path.
/software/astro/src/{user}
This space is limited so please be careful with the size of the code (e.g. people downloading all branches/tags in svn repositories).

Each user has an environment space (for python virtual environments, see python_env_url for more details) at the following path:
/software/astro/env/{user}

'''Any location not included in the former paths is not allowed and its contents erased on sight.'''

== Download code and git rules ==

These are the git rules for developers at PIC.

The methodology written below is a try to help the code development of the team and they are thought for non-experts git users.

It has been compiled from the official git documentation, which we strongly recommend to look at (at least the first three chapters):

[https://git-scm.com/book/en/v2 Git documentation]

And from this git branch model:

[http://nvie.com/posts/a-successful-git-branching-model A successful Git branching model]

We assume you already have a PIC account.

Codes are hosted at https://gitlab.pic.es.

1. Download the code (first you need to have permissions to do it)

The user software area can only be edited from services.astro so you must have access to this machine in order to edit the code.

1.1. Access ui:

ssh {user}@ui.pic.es

1.2. Access services.astro:

ssh {user}@ui.pic.es

1.3 Go to your software area in which you are going to develop your codes:

cd /software/astro/src/{user}/

1.4 Copy the code from the gitlab repository:

git clone https://gitlab01.pic.es/{project}/{pipeline}.git

2. Create your own branch

Every project has two main protected branches: ''master'' and ''develop''. Protected means you, as a standard developer, will not have permissions to write on them. Therefore in order to develop your features in the code you need to create your own branch that will always come from the ''develop'' branch.

2.1 Enter in the project directory:

cd {pipeline}

2.2 Create the branch:

git checkout -b feature_branch_name origin/develop

3. Modify the code

3.1 Day Tip:

Everyday you sit in your computer and want to modify the code, in order not to be outdated in the changes made in the develop branch, you should do:

3.1.1 Download any modification in the code

git fetch

3.1.2 Incorporate changes in the ''develop'' branch into your feature_branch_name branch

git rebase origin/develop

(Hopefully there will be no conflicts if all developers are working in ''independent'' branches. If this is not the case and you have doubts after reading the references given in point 5 below, please call us before meshing it up!)

3.2 See the changes you have done

git status

3.3 Add changes

git add changed_files

3.4 Commit changes

git commit -m "message describing the modifications"

4. Finish the new feature

Once you finish to develop, debug and test the new feature you send us an email.

We immediately will send you back another one saying that your feature has been integrated into the develop branch.

Note that your branch will be deleted.

5. Incorporate changes and start a new feature again

In order to start with a new feature you need to incorporate the changes we just did (integrate the feature into develop) and create another new branch:

git fetch

git checkout -b anoter_feature_branch_name origin/develop

=== Create new environment ===

Go to the directory in which you want to create a virtual environment.

{user_dir}/env/

These are the commands to create, activate and set up the platform of binary python wheel packages:

cd {user_dir}/env/
/software/astro/sl6/python/2.7.11/bin/python /software/astro/sl6/python/2.7.11/bin/virtualenv {virtual_env_name}
. {virtual_env_name}/bin/activate
# Define _PYTHON_HOST_PLATFORM
export _PYTHON_HOST_PLATFORM=pic_sl6

Then, having the virtual environment activated, in order to install any package ({package}) you just have to type:

pip install {package}

==== Example : paudm environment ====

With the virtual environment created and activated as described above, install the paudm packages.

For each package (in order: tools, resources, environment, register, nightly,...) run:
cd /software/astro/src/{user}/{paudm_code_dir}
pip install -e .

The dependencies written in the setup.py will be fulfilled.

AC UserManual

2017-01-04T09:59:53Z

Tonello: /* Create new environment */

== Storage ==
=== Massive storage ===
Each project has a massive storage space accessible at the following path:
/pnfs/pic.es/data/astro/{project} (see projects_url for more details)
which has only read permissions for the project's users.

Inside the directory there are two different paths corresponding to two different back-ends:

==== Tape ====
/pnfs/pic.es/data/astro/{project}/tape
As its name suggests, the data in the tape path is stored in magnetic tapes, and is critical, such as raw data or very difficult data to obtain or to get.
The size of each file is usually large, from 1-2GB to 100-200GB, due to technical reasons (they are usually ''iso'' or ''tar.bz2'' files).
Data in tapes is not very often accessed. It is necessary a previous step when accessing data in tapes, which is the pre-stage. This step is the action of reading the magnetic tape and put the data into a buffer. Once the data is in the buffer one can access the data as it was in a disk. After some time, the data is removed from the buffer. In case the data in tape has to be available for long time get in touch with the contant person of the project.

==== Disk ====
/pnfs/pic.es/data/astro/{project}/disk
Disk data is usually the data being currently used by the project, and it is being very often accessed. The size of the files is not important here.

=== Scratch ===

Each user has a scratch space at the following path:
/nfs/astro/{user}
This space is thought as a sandbox.
When the project decides some data from any scratch space is important enough for the whole collaboration, the contact person for each project will move the data into the /pnfs storage.

Take note that all '''data older than 6 months may be erased at any time''' without prior notice.

=== Software ===
Each user has a software space (for editable source code) at the following path.
/software/astro/src/{user}
This space is limited so please be careful with the size of the code (e.g. people downloading all branches/tags in svn repositories).

Each user has an environment space (for python virtual environments, see python_env_url for more details) at the following path:
/software/astro/env/{user}

'''Any location not included in the former paths is not allowed and its contents erased on sight.'''

== Download code and git rules ==

These are the git rules for developers at PIC.

The methodology written below is a try to help the code development of the team and they are thought for non-experts git users.

It has been compiled from the official git documentation, which we strongly recommend to look at (at least the first three chapters):

[https://git-scm.com/book/en/v2 Git documentation]

And from this git branch model:

[http://nvie.com/posts/a-successful-git-branching-model A successful Git branching model]

We assume you already have a PIC account.

Codes are hosted at https://gitlab.pic.es.

1. Download the code (first you need to have permissions to do it)

The user software area can only be edited from services.astro so you must have access to this machine in order to edit the code.

1.1. Access ui:

ssh {user}@ui.pic.es

1.2. Access services.astro:

ssh {user}@ui.pic.es

1.3 Go to your software area in which you are going to develop your codes:

cd /software/astro/src/{user}/

1.4 Copy the code from the gitlab repository:

git clone https://gitlab01.pic.es/{project}/{pipeline}.git

2. Create your own branch

Every project has two main protected branches: ''master'' and ''develop''. Protected means you, as a standard developer, will not have permissions to write on them. Therefore in order to develop your features in the code you need to create your own branch that will always come from the ''develop'' branch.

2.1 Enter in the project directory:

cd {pipeline}

2.2 Create the branch:

git checkout -b feature_branch_name origin/develop

3. Modify the code

3.1 Day Tip:

Everyday you sit in your computer and want to modify the code, in order not to be outdated in the changes made in the develop branch, you should do:

3.1.1 Download any modification in the code

git fetch

3.1.2 Incorporate changes in the ''develop'' branch into your feature_branch_name branch

git rebase origin/develop

(Hopefully there will be no conflicts if all developers are working in ''independent'' branches. If this is not the case and you have doubts after reading the references given in point 5 below, please call us before meshing it up!)

3.2 See the changes you have done

git status

3.3 Add changes

git add changed_files

3.4 Commit changes

git commit -m "message describing the modifications"

4. Finish the new feature

Once you finish to develop, debug and test the new feature you send us an email.

We immediately will send you back another one saying that your feature has been integrated into the develop branch.

Note that your branch will be deleted.

5. Incorporate changes and start a new feature again

In order to start with a new feature you need to incorporate the changes we just did (integrate the feature into develop) and create another new branch:

git fetch

git checkout -b anoter_feature_branch_name origin/develop

=== Create new environment ===

There is a directory in which each user can create its own virtual environments.

/software/astro/env/{user}/

These are the commands to create, activate and set up the platform of binary python wheel packages:

cd /software/astro/env/{user}/
/software/astro/sl6/python/2.7.11/bin/python /software/astro/sl6/python/2.7.11/bin/virtualenv {virtual_env_name}
. {virtual_env_name}/bin/activate
# Define _PYTHON_HOST_PLATFORM
export _PYTHON_HOST_PLATFORM=pic_sl6

Then, having the virtual environment activated, in order to install any package ({package}) you just have to type:

pip install {package}

==== Example : paudm environment ====

With the virtual environment created and activated as described above, install the paudm packages.

For each package (in order: tools, resources, environment, register, nightly,...) run:
cd /software/astro/src/{user}/{paudm_code_dir}
pip install -e .

The dependencies written in the setup.py will be fulfilled.