Public PIC Wiki - Recent changes [en]

Faq

Tallada — Wed, 15 Apr 2026 15:05:17 GMT

Main Page

Tallada — Wed, 15 Apr 2026 13:42:19 GMT

Faq

Tallada — Wed, 15 Apr 2026 13:32:08 GMT

Main Page

Tallada — Wed, 15 Apr 2026 13:18:36 GMT

Lost OTP

Msantama — Mon, 13 Apr 2026 08:49:55 GMT

Created page with " = OTP Recovery Guide (Lost OTP Device) = If you lose access to your OTP device, follow these steps to recover your accou..."

New page

= OTP Recovery Guide (Lost OTP Device) =

If you lose access to your OTP device, follow these steps to recover your account and configure a new OTP method.

# On the Keycloak login page, click '''Forgot password / OTP'''.
# Enter your username. You should receive a recovery email in your inbox.
# Open the email and follow the recovery link.
# Scan the new QR code with your authenticator app.
# Enter the generated OTP code and a device name.
#* The device name is mandatory.
# Change your password when prompted.
#* This is required for security reasons.
# Log in again as usual.
#* On the OTP selection page, you will see two devices.
#* Select the device with the name you just entered.
#* The old one will usually appear as '''unnamed''' (if no name was set previously).


== Recommended Cleanup ==

After logging in, go to:

https://idp.pic.es/realms/PIC/account/account-security/signing-in

Then delete the previous OTP device. If it had no custom name, it may appear as '''otp'''.

Main Page

Msantama — Mon, 13 Apr 2026 08:49:12 GMT

Getting started

Spark on Jupyter

Tallada — Wed, 08 Apr 2026 08:51:24 GMT

Created page with "= Accessing PIC Big Data Services from JupyterLab = PIC's Jupyter service is integrated with the Big Data platform. This allows you to access services such as '''HDFS''', '''..."

New page

= Accessing PIC Big Data Services from JupyterLab =

PIC's Jupyter service is integrated with the Big Data platform. This allows you to access services such as '''HDFS''', '''Hive''', and '''Spark''' directly from your JupyterLab environment.

== Requirements ==

=== 1. Kerberos Authentication ===

You must have a '''valid Kerberos ticket''' before accessing any Big Data service.

==== Step-by-step ====

# Open a terminal session on the same node where your Jupyter session is running.
#* In JupyterLab, use the ''Launcher'' → ''Terminal''.
# Run the following commands:

<pre>
kinit -n -c ~/.fast.ccache @PIC.ES
kinit -T ~/.fast.ccache
</pre>

==== Verify your ticket ====

<pre>
klist
</pre>

You should see a valid ticket with a non-expired timestamp.

==== Notes ====

* If the ticket expires, you must repeat the process.
* Without a valid Kerberos ticket, access to HDFS, Hive, or Spark will fail.

==== Quick test (HDFS access) ====

<pre>
hdfs dfs -ls /
</pre>

If this works, your authentication is correctly configured.

---

=== 2. Python Environment ===

You need specific Python packages depending on the service:

{| class="wikitable"
|-
! Service !! Required packages
|-
| Spark || findspark, pyspark
|-
| Hive || pyhive[hive_pure_sasl, kerberos]
|}

==== Installation example ====

<pre>
pip install findspark pyspark
pip install "pyhive[hive_pure_sasl, kerberos]"
</pre>

---

== Using Spark from Jupyter ==

=== Setup ===

<pre>
import os

# Environment configuration
os.environ["HADOOP_HOME"] = "/usr/local/hadoop"
os.environ["HADOOP_CONF_DIR"] = "/usr/local/hadoop/etc/hadoop"
os.environ["HIVE_HOME"] = "/usr/local/hive"
os.environ["HIVE_CONF_DIR"] = "/usr/local/hive/conf"

import findspark
findspark.init()
</pre>

=== Create a Spark session ===

<pre>
from pyspark.sql import SparkSession

spark = (
SparkSession.builder
.appName("example")
.enableHiveSupport()
.getOrCreate()
)

sc = spark.sparkContext
</pre>

=== Example: Read a Hive table ===

<pre>
df = spark.sql("SELECT * FROM some_database.some_table LIMIT 10")
df.show()
</pre>

---

== Using Hive (PyHive) from Jupyter ==

=== Connect to Hive ===

<pre>
from pyhive import hive

conn = hive.connect(
host="hsrv01.pic.es",
port=10000,
kerberos_service_name="hive",
auth="KERBEROS",
)

cursor = conn.cursor()
</pre>

=== Execute a query ===

<pre>
query = "SELECT * FROM some_database.some_table LIMIT 10"
cursor.execute(query)

rows = cursor.fetchall()
colnames = [c[0] for c in cursor.description]
</pre>

=== Convert to Astropy Table (optional) ===

<pre>
from astropy.table import Table

table = Table(rows=rows, names=colnames)
</pre>

---

== Troubleshooting ==

=== Kerberos errors ===
* Run <code>klist</code> and check expiration
* Re-run <code>kinit</code> if needed

=== Hive connection fails ===
* Ensure Kerberos ticket is valid
* Verify correct host (<code>hsrv01.pic.es</code>)
* Check that required Python packages are installed

=== Spark does not start ===
* Verify environment variables (<code>HADOOP_HOME</code>, <code>HIVE_HOME</code>)
* Ensure <code>findspark.init()</code> is executed before creating the session

---

== Summary ==

# Obtain a Kerberos ticket (<code>kinit</code>)
# Install required Python packages
# Configure environment variables
# Use Spark or Hive from your notebook

---

For further assistance, contact PIC support (user.support@pic.es).

Main Page

Tallada — Wed, 08 Apr 2026 08:17:49 GMT

Distributed computing

JupyterHub

Torradeflot — Wed, 08 Apr 2026 08:14:19 GMT

GPUs

File:GPU utilization 20260408.png

Torradeflot — Wed, 08 Apr 2026 08:11:29 GMT

Torradeflot uploaded File:GPU utilization 20260408.png

New page

JupyterHub

Torradeflot — Wed, 08 Apr 2026 08:10:10 GMT

GPUs

HTCondor

Cacosta — Fri, 27 Mar 2026 13:26:05 GMT

The htcondor command

Gitlab

Torradeflot — Mon, 23 Mar 2026 09:29:51 GMT

Best practices / recommendations

@@ Line 33: / Line 33: @@
 Follow these steps to recover your account and configure a new OTP token.
-# On the Keycloak login page, click '''Forgot password / OTP'''.
+# On the login page, click '''Forgot password / OTP''' or go [https://www.pic.es/reset here].
 # Enter your username. You should receive a recovery email in your inbox.
 # Open the email and follow the recovery link.
@@ Line 47: / Line 47: @@
 Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].

@@ Line 2: / Line 2: @@
 * [[PIC description|Introduction to PIC]]
 * [[PIC account|Get a PIC account]]
 * [[PIC_User_Manual | User manual]]
 * [[faq| Frequently asked questions]]

@@ Line 23: / Line 23: @@
 * [[HDFS Access via VOSpace]]
 * [[Transferring data to/from PIC]]
 === Other services ===
@@ Line 31: / Line 29: @@
 == Experiments ==
 * [[Euclid]]
 * [[AGN ICE]]
 * [[ICFO]]

@@ Line 2: / Line 2: @@
 * [[PIC description|Introduction to PIC]]
 * [[PIC account|Get a PIC account]]
 * [[PIC_User_Manual | User manual]]
 * [[faq| Frequently asked questions]]

@@ Line 14: / Line 14: @@
 * [[Dask]]
 * Spark:
-** [[Spark on Hadoop|on Hadoop]]
+** [[Spark on Jupyter|on Jupyter]]
 ** [[Spark_on_farm|on HTCondor]]

← Older revision		Revision as of 13:32, 15 April 2026
(One intermediate revision by the same user not shown)
Line 5:		Line 5:
	== Can an undergraduate student in my group have an account? ==		== Can an undergraduate student in my group have an account? ==
	Yes. Undergraduate students can have PIC accounts without any problem.		Yes. Undergraduate students can have PIC accounts without any problem.
		+
		+	== How do I get a Kerberos token? ==
		+	A valid token is needed to interface with some PIC services, like the Hadoop platform (HDFS, Hive, Spark, ...) or CTA.
		+	Tokens are issued for the machine in which they are requested, if you hop to another machine, you may need to get another token.
		+
		+	In order to get a token, run the following commands on a terminal.
		+	If you are on JupyterLab, use ''Launcher'' → ''Terminal''.
		+
		+	<pre>
		+	kinit -n -c ~/.fast.ccache @PIC.ES
		+	kinit -T ~/.fast.ccache
		+	</pre>
		+
		+	You could also define an alias for your own convenience:
		+
		+	<pre>
		+	alias kinit="kinit -n -c ~/.fast.ccache @PIC.ES; kinit -T ~/.fast.ccache"
		+	</pre>
		+
		+	Optionally, you can check the presence and expiration date of your token using:
		+
		+	<pre>
		+	klist
		+	</pre>
		+
		+	== I lost access to my OTP, how can I reset it? ==
		+	Follow these steps to recover your account and configure a new OTP token.
		+
		+	# On the Keycloak login page, click '''Forgot password / OTP'''.
		+	# Enter your username. You should receive a recovery email in your inbox.
		+	# Open the email and follow the recovery link.
		+	# Scan the new QR code with your authenticator app.
		+	# Enter the generated OTP code and a device name.
		+	#* The device name is mandatory.
		+	# Change your password when prompted.
		+	#* This is required for security reasons.
		+	# Log in again as usual.
		+	#* On the OTP selection page, you will see two devices.
		+	#* Select the device with the name you just entered.
		+	#* The old one will usually appear as '''unnamed''' (if no name was set previously).
		+
		+	Afterwards, we recommend removing the old OTP factor from the [https://idp.pic.es/realms/PIC/account/account-security/signing-in user profile section].

@@ Line 404: / Line 404: @@
      GPU-a2361e9d-c520-684b-b21b-60cc3a59b05b
-You can then track the GPU utilization (memory and load) using the GPU Dashboard in the left sidebar
+You can track the GPU utilization (memory and load) using the GPU Dashboard in the left sidebar
 == Code samples ==

@@ Line 399: / Line 399: @@
 == GPUs ==
-The way to identify the GPUs that are assigned to your job is:
+The GPUs that are assigned to your job are listed in the environment variable CUDA_VISIBLE_DEVICES. In a terminal run "echo $CUDA_VISIBLE_DEVICES". The environment variable contains a list of comma-separated GPU ids. With this you will already know how many gpus are assigned to your job. If the variable does not exist, there are no gpus assigned to the job
-* list the gpus with nvidia-smi, in a terminal run "nvidia-smi -L", and look for the gpus you've been assigned. Remember their indexes (integers from 0 to 7)
+    [torradeflot@gpu05 ~]$ echo $CUDA_VISIBLE_DEVICES
+    GPU-a2361e9d-c520-684b-b21b-60cc3a59b05b
 == Code samples ==

← Older revision		Revision as of 13:26, 27 March 2026
(8 intermediate revisions by the same user not shown)
Line 1,415:		Line 1,415:

	Or you can remove all your jobs just using the "-all" option.		Or you can remove all your jobs just using the "-all" option.
		+
		+	= The htcondor command =
		+
		+	In newer HTCondor versions, a new unified command, htcondor, has been introduced to help you manage your jobs.
		+
		+	<pre>
		+	$ htcondor -h
		+	usage: htcondor [-h] [-v] [-V] [-q] {dag,job,jobset,eventlog,credential,server,ap,cm,ocu,annex} ...
		+
		+	A tool for managing HTCondor jobs and resources.
		+
		+	optional arguments:
		+	-h, --help show this help message and exit
		+	-v, --verbose Increase verbosity of output (can be specified multiple times)
		+	-V, --version Print the HTCondor version and exit
		+	-q, --quiet Decrease verbosity of output (can be specified multiple times)
		+
		+	subcommands:
		+
		+	{dag,job,jobset,eventlog,credential,server,ap,cm,ocu,annex}
		+	Types of HTCondor objects that can be managed
		+	</pre>
		+
		+	A complete man page is available via the command line or at the following URL:
		+
		+	https://htcondor.readthedocs.io/en/latest/man-pages/htcondor.html#index-1
		+
		+	The goal of the htcondor command is to consolidate the functionality of multiple legacy commands into a single, intuitive interface. HTCondor developers have insisted that the old commands (like condor_submit, condor_q, etc.) are never going to disappear. However, you might find the way this new command interacts with the system and presents information to be quite useful and user-friendly.
		+
		+	Although the command contains different subcommands, as a standard user, the three main subcommands are <b>job</b>, <b>eventlog</b>, and <b>dag</b>.
		+
		+	== Managing jobs ==
		+
		+	You can easily submit a job:
		+
		+	<pre>
		+	$ htcondor job submit test.sub
		+	Job 1522136 was submitted.
		+	</pre>
		+
		+	And query its status to get detailed information. For example, after submission, it might show as idle:
		+
		+	<pre>
		+	$ htcondor job status 1522136
		+	Job 1522136 is currently idle.
		+	It was submitted 11.100039 seconds ago.
		+	It requested 1.000 GB of memory.
		+	It requested 15.000 GB of disk space.
		+	Job has 0.0% goodput for 0.0 second of wall clock time.
		+	</pre>
		+
		+	Once the job starts to run, the output updates automatically to reflect its running state:
		+
		+	<pre>
		+	$ htcondor job status 1522136
		+	Job 1522136 is currently running.
		+	It started running 5.665923 seconds ago.
		+	It was submitted 17.66585 seconds ago.
		+	Its current disk usage is 1.000 KB out of 15.000 GB requested.
		+	Job has 0.0% goodput for 0.0 second of wall clock time.
		+	</pre>
		+
		+	It is also very convenient to query the log, the standard output, and standard error directly from the command line (like using condor_tail).
		+
		+	<pre>
		+	$ htcondor job out 1522136.0 \| tail -3
		+	stress: info: [15] dispatching hogs: 16 cpu, 0 io, 0 vm, 0 hdd
		+	stress: info: [15] successful run completed in 600s
		+	</pre>
		+
		+	<pre>
		+	$ htcondor job log 1522136.0 \| tail -4
		+	1 - MemoryUsage of job (MB)
		+	1012 - ResidentSetSize of job (KB)
		+	...
		+	</pre>
		+
		+	You can check the status of a completed job that is no longer in the active queue. HTCondor will automatically search the history:
		+
		+	<pre>
		+	$ htcondor job status 1522136.0
		+	Job 1522136.0 was not found in the active queue, checking history...
		+	(This may take a while, Ctrl-C to cancel.)
		+	Job 1522136.0 has completed.
		+	It completed 4 hours ago.
		+	It was submitted 4 hours ago.
		+	Its last memory usage was 2.000 MB out of 1.000 GB requested.
		+	Its last disk usage was 100.000 KB out of 15.000 GB requested.
		+	Job had 100.0% goodput for 10 minutes of wall clock time.
		+	</pre>
		+
		+	You should use <b>--skip-history</b> to avoid this behaviour.
		+
		+	Furthermore, the command provides information about the held jobs, telling you the reason for the hold:
		+
		+	<pre>
		+	$ htcondor job status 1522207.0
		+	Job 1522207.0 is currently held.
		+	It has been held for 1 minute.
		+	It was held because "Your job memory 8367MB exceeded the Job Memory Limit 1.280000000000000E+03MB".
		+	It has been held 3 times.
		+	It was submitted 2 minutes ago.
		+	Its last memory usage was 9.537 GB out of 512.000 MB requested.
		+	Its last disk usage was 100.000 KB out of 15.000 GB requested.
		+	HTCondor has attempted to start the job 1 time.
		+	The job has started 1 time.
		+	Job has 0.0% goodput for 9.0 seconds of wall clock time.
		+	</pre>
		+
		+	== Eventlog ==
		+
		+	The htcondor command is useful for reading and understanding the raw log files.
		+
		+	You can use the <i>follow</i> subcommand (similar to <i>tail -f</i>) to see how your job evolves, presented in tabulated format:
		+
		+	<pre>
		+	$ htcondor eventlog follow test.log
		+	Job Host Start Time Evict Time Evictions Wall Time Good Time CPU Usage
		+	1522209.0 193.109.174.74 0
		+	Job Host Start Time Evict Time Evictions Wall Time Good Time CPU Usage
		+	1522209.0 slot1_43@tds515.pic.es 3/27/26 13:56 0
		+	Job Host Start Time Evict Time Evictions Wall Time Good Time CPU Usage
		+	1522209.0 slot1_43@tds515.pic.es 3/27/26 13:56 0
		+	Job Host Start Time Evict Time Evictions Wall Time Good Time CPU Usage
		+	1522209.0 slot1_43@tds515.pic.es 3/27/26 13:56 3/27/26 13:57 0 0+00:01:00 0+00:01:00 0+00:01:58
		+	</pre>
		+
		+	Or you can perform a direct query to read the final state of the file:
		+
		+	<pre>
		+	$ htcondor eventlog read test.log
		+	Job Host Start Time Evict Time Evictions Wall Time Good Time CPU Usage
		+	1522209.0 slot1_43@tds515.pic.es 3/27/26 13:56 3/27/26 13:57 0 0+00:01:00 0+00:01:00 0+00:01:58
		+	</pre>
		+
		+
		+	== Managing DAGs ==
		+
		+	The new htcondor command is especially useful for monitoring and understanding how your DAG submissions are progressing.
		+
		+	You can submit a DAG just as easily as a regular job:
		+
		+	<pre>
		+	$ htcondor dag submit test.dag
		+	DAG 1522211 was submitted.
		+	</pre>
		+
		+	Once submitted, you can check its status. The output provides a summary of the running and waiting nodes, along with a visual progress bar indicating how much work is left to finish:
		+
		+	<pre>
		+	$ htcondor dag status 1522211.0
		+	DAG 1522211.0 [test.dag] has been running for and 00:00:34
		+	DAG has submitted 1 job(s), of which:
		+	1 is submitted and waiting for resources.
		+	DAG contains 2 node(s) total, of which:
		+	[=] 1 is running: 1 job.
		+	[-] 1 is waiting on other nodes to finish.
		+	DAG is running normally.
		+	[===============================================-----------------------------------------------] DAG is 0.00% complete.
		+	</pre>

	= From Torque to HTCondor =		= From Torque to HTCondor =

← Older revision		Revision as of 09:29, 23 March 2026
Line 45:		Line 45:
	* '''Do not track jupyter notebooks (.ipynb)''' pair them with a script using [[JupyterHub#jupytext\|jupytext]]		* '''Do not track jupyter notebooks (.ipynb)''' pair them with a script using [[JupyterHub#jupytext\|jupytext]]
	* '''Do not upload confidential data''' passwords, ssh keys, etc will be there forever		* '''Do not upload confidential data''' passwords, ssh keys, etc will be there forever
		+
		+	== Troubleshooting ==
		+
		+	=== Errors in CI/CD jobs ===
		+
		+	* ''pods XXXX is forbidden: exceeded quota'': There's a limit on the amount of resources that can be used simultaneously for CI/CD jobs. This takes into account your own jobs and jobs from other projects. You should be able to retry the pipeline after some period to wait for resources to be freed. If it doesn't work contact the administrator.
		+	* ''ERROR: Preparation failed: couldn't prepare overwrites: invalid build requests specified: the resource "XXXXXX" requested "X" is higher than limit allowed "X"'': Your job requested more than the allowed resources. Lower down the amount of resources requested.
		+	* ''ERROR: Job failed (system failure): Error in container build: exit code: 137, reason: 'OOMKilled''': The job exceeded the memory limit and it was killed. Try increasing the memory requested for you CI/CD job. If this is not enough contact the servicea administrator.
		+
		+	=== Configure resource limits in CI/CD jobs ===
		+
		+	Gitlab CI/CD jobs run as pods on one of PIC's kubernetes clusters. Pods can have resource requests and limits. Resource requests are effectively allocated for the pod, which will be killed if it exceeds the resource limits.
		+
		+	Default resources are:
		+	* 1GiB memory and 0.5 cpu requested
		+	* 2GiB memory and 1 cpu limit
		+
		+	Default resources can be overwritten by setting environment variables in the job configuration in .gitlab-ci.yml like this:
		+
		+	cicd_job:
		+	image: python:3.11-slim
		+	variables:
		+	KUBERNETES_CPU_REQUEST: "1"
		+	KUBERNETES_CPU_LIMIT: "2"
		+	KUBERNETES_MEMORY_REQUEST: "4Gi"
		+	KUBERNETES_MEMORY_LIMIT: "6Gi"
		+
		+	Maximum resources that can be overwritten are:
		+	* 6 GiB memory and 2 cpu requested
		+	* 8 GiB memory and 4 cpu limit