Public PIC Wiki - User contributions [en]

HDFS Access via VOSpace

2025-07-23T11:30:50Z

Sbogaart: Created page with "= Introduction = PIC provides access to the distributed file system HDFS through a '''VOSpace server''', following the IVOA standard described in [https://www.ivoa.net/docume..."

= Introduction =

PIC provides access to the distributed file system HDFS through a '''VOSpace server''', following the IVOA standard described in [https://www.ivoa.net/documents/VOSpace/20180502/PR-VOSpace-2.1-20180502.html VOSpace 2.1].

This service is an alternative to WebDAV access, allowing users to manage their data programmatically and in a structured way using tools compatible with the Virtual Observatory (VO) ecosystem. It is especially aimed at users who require data management operations such as reading, writing, moving, and metadata querying within a standardized environment.

= How to connect to the service =

== VOSpace Endpoint ==

The VOSpace server is available at the following URL:

https://vospace.pic.es/vospace

== Compatible clients ==

You can access the service using tools compatible with the VOSpace 2.1 standard:

* [https://pypi.org/project/vos/ '''vos''' client by CADC]
: Install with pip:
pip install vos>=3.6.3

* '''curl''':
: Can be used to perform HTTP operations following the examples defined in the [https://www.ivoa.net/documents/VOSpace/20180502/PR-VOSpace-2.1-20180502.html VOSpace 2.1 standard], including node creation, file transfers, and property queries.

== Authentication ==

The VOSpace server allows both anonymous and authenticated access.
To access restricted data or personal space, users must specify their '''PIC username and password'''.

= Usage Guidelines =

Once connected to the service, users can perform a variety of operations defined by the VOSpace standard. Supported functionalities include:

* '''getProtocols''': Query supported transfer protocols.
* '''getViews''': View available data formats (views).
* '''getProperties''': Obtain node properties.
* '''createNode''': Create files or folders.
* '''getNode''': Retrieve node information.
* '''deleteNode''': Delete files or folders.
* '''moveNode''': Move nodes inside the VOSpace.
* '''put / get''': Upload or download files.
* '''pushToVoSpace / pullFromVoSpace''': Transfer files via external URLs.

= Best Practices =

* Avoid using the '''!''' character in file or folder names, as it may cause compatibility issues with the CADC '''vos''' client.
* Use tools that comply with the VOSpace standard to ensure compatibility and avoid transfer errors.

= Troubleshooting =

'''Cannot access the server:'''
* Verify the URL is correct: <code>https://vospace.pic.es/vospace</code>
* Make sure PIC credentials are set correctly.
* Check client configuration (e.g., '''vos''', '''curl''').

'''Problems with properties:'''
* The '''setProperties''' operation is '''not allowed''' due to permission restrictions on HDFS.

= Security and Data Management =

* Protect PIC credentials. Do not share or store them in plain text.
* For automation, consider using secure credential managers or temporary storage.

= Usage Examples =

== Using the CADC '''vos''' client (Python) ==

This server supports [https://pypi.org/project/vos/ the CADC vos client] and operations like '''listdir''', '''mkdir''', '''copy''', '''move''', and '''delete'''. Minimum version: '''vos >= 3.6.3'''.

=== Configuration ===

* Create a config file (e.g. <code>vos-config.ini</code>):
[vos]
resourceID = https://vospace.pic.es/vospace http

* Export the file path as an environment variable:

export VOSPACE_CONFIG_FILE=/path/to/vos-config.ini

* Create <code>~/.netrc</code> for authentication:

machine https://vospace.pic.es/vospace
login USERNAME
password PASSWORD

* Set permissions:

chmod 600 ~/.netrc
=== Python usage examples ===

import vos

client = vos.Client()

# List contents
files = client.listdir("https://vospace.pic.es/vospace/nodes/user/my_user/")
print("Contents:", files)

# Create directory
client.mkdir("https://vospace.pic.es/vospace/nodes/user/my_user/mydir")

# Upload file
client.copy("localfile.txt", "https://vospace.pic.es/vospace/user/my_user/mydir/localfile.txt")

# Download file
client.copy("https://vospace.pic.es/vospace/user/my_user/mydir/localfile.txt", "downloaded.txt")

# Move file
client.move(
"https://vospace.pic.es/vospace/user/my_user/mydir/localfile.txt",
"https://vospace.pic.es/vospace/user/my_user/mydir/renamedfile.txt"
)

# Get node info
node = client.get_node("https://vospace.pic.es/vospace/nodes/user/my_user/mydir/renamedfile.txt")
print("Properties:", node.props)

# Delete resource
client.delete("https://vospace.pic.es/vospace/nodes/user/my_user/mydir")

'''Note''': Ensure configuration is properly set.

== Using curl ==

You can also interact with the server using '''curl''' or other HTTP tools.

These must follow the [https://www.ivoa.net/documents/VOSpace/20180502/PR-VOSpace-2.1-20180502.html VOSpace 2.1] specification.

'''Important:'''
* Avoid using '''!''' in URLs.
* Use full HTTP paths like <code>/vospace/nodes/...</code>.

= VOSpace Server Endpoints =

The server provides the following REST endpoints per the VOSpace 2.1 specification:

{| class="wikitable"
! Method !! Path !! Description
|-
| GET || <code>/vospace/protocols</code> || Retrieves supported transfer protocols
|-
| GET || <code>/vospace/views</code> || Retrieves available data views
|-
| GET || <code>/vospace/properties</code> || Retrieves node properties
|-
| GET || <code>/vospace/capabilities</code> || Retrieves server capabilities
|-
| GET || <code>/vospace/{job_id}</code> || Retrieves job information
|-
| GET || <code>/vospace/{job_id}/phase</code> || Retrieves job phase
|-
| GET || <code>/vospace/{job_id}/error</code> || Retrieves job error info
|-
| GET || <code>/vospace/{job_id}/results/transferDetails</code> || Retrieves transfer details
|-
| GET || <code>/vospace/{path}</code> || Retrieves a file/folder node
|-
| GET || <code>/vospace/{path}</code> || Downloads a file (streaming)
|-
| PUT || <code>/vospace/{path}</code> || Creates a file or folder
|-
| PUT || <code>/vospace/{path}</code> || Uploads a file
|-
| POST || <code>/vospace/</code> || Moves or copies a node (returns 303 redirect)
|-
| POST || <code>/vospace/synctrans</code> || Push/pull transfer
|-
| POST || <code>/vospace/{path}</code> || Set properties (''not supported'')
|-
| POST || <code>/vospace/{job_id}/phase</code> || Change job phase
|-
| DELETE || <code>/vospace/{path}</code> || Deletes a file or folder
|}

== Notes ==
* <code>{path}</code> and <code>{job_id}</code> are dynamic parameters.
* The server uses standard HTTP error codes: 400, 403, 404, 409, 500.
* '''PUT''' supports uploads and node creation.
* '''POST /vospace/''' moves or copies nodes (responds with '''303 See Other''').
* '''setProperties''' is disabled due to HDFS permissions.
* File downloads return streamed content with '''GET'''.
* Avoid using '''!''' in paths for full compatibility with the CADC client.

Main Page

2025-07-23T10:50:15Z

Sbogaart: /* Storage */

== Getting started ==
* [[PIC description|PIC in an image]]
* [[PIC account|Get a PIC account]]
* [[PIC_User_Manual | User manual]]
* [[faq| Frequently asked questions]]

== Services ==
=== Distributed computing ===
* [[HTCondor]]
* [[Dask]]
* Spark:
** [[Spark on Hadoop|on Hadoop]]
** [[Spark_on_farm|on HTCondor]]

=== Storage ===
* [[Storage]]
* [[Hadoop Distributed File System (HDFS)]]
* [[HDFS Access via VOSpace]]
* [[Transferring data to/from PIC]]

=== User interfaces ===
* [[Login machines]]
* [[JupyterHub]]

=== Other services ===
* [[Gitlab]]
* [[CosmoHub]]

== Experiments ==

* [[Euclid]]
* [[AGN ICE]]
* [[ICFO]]

== More technical information ==

* [[Storage Department]]

Hadoop Distributed File System (HDFS)

2025-05-19T08:41:00Z

Sbogaart: /* How to connect to the service */

= Introduction =

PIC offers a WebDAV service over WebHDFS for seamless file transfer and management. This service is designed for users who need to upload, download, and manage large files on HDFS through a familiar file management interface. The WebDAV protocol allows users to access the HDFS in a way that mimics the experience of working with local file systems, offering the flexibility and ease of use of tools like Finder, File Explorer, and rclone, among others.

WebDAV-compatible clients allow for the management of large datasets and offer multithreaded operations for enhanced performance. This makes the WebDAV service ideal for handling large-scale data uploads and downloads, with support for efficient file transfers.

The service also provides a simple and effective way to interface with HDFS, especially for users who prefer a file-system-like experience for managing their data, rather than relying on more technical methods like command-line tools.

= How to connect to the service =

To connect to the WebDAV service, follow the steps below:

# '''Use a WebDAV-compatible client:'''
#*Finder (macOS)
#*File Explorer (Windows)
#*Linux File System (Linux)
#*rclone (CLI)
#*Cyberduck
#*CrossFTP
#*curl
# '''Mount the WebDAV endpoint:'''
#: The WebDAV server is accessible via the following URL: https://webdav-hdfs.pic.es/ [https://webdav.pic.es/]
# '''Authenticate with your PIC credentials:'''
#: When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.
# '''Browse, upload, and download files:'''
#: Once connected, you will be able to manage your files in the same way you would with any local file system. You can drag and drop files, create directories, and manage large datasets directly on the HDFS.
**''Tip'': Using a multithreaded client like rclone or Cyberduck will help improve upload and download speeds for large files.

= Usage Guidelines=

* File Uploads:

** The WebDAV service allows you to upload files of any size to HDFS.
** For large datasets, it is recommended to use a multithreaded client, as this will optimize the upload process and make it faster and more efficient.

* File Downloads:

** You can also download files from the HDFS to your local machine.
** Multithreading is supported, allowing you to download large files quickly.

* File Management:

** Files and directories can be created, deleted, or renamed directly within your WebDAV-compatible client.

** You can also move files around within your HDFS storage, making file management easier.

=Best Practices=
'''Optimize large file transfers:'''

: To improve performance, use tools like rclone or Cyberduck for multithreaded file transfers. This helps manage the upload and download of large files or large quantities of files more efficiently.

=Troubleshooting=

'''Cannot Connect to the WebDAV Service:'''
:If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.

'''Slow Uploads or Downloads:'''
:If you experience slow file transfer speeds, check if your client supports multithreaded transfers and consider using a tool like rclone or Cyberduck, which support multiple threads for faster transfers.

'''File Uploads Fail'''
:If file uploads fail, try splitting large files into smaller chunks or use an alternative tool that supports better error handling and retry capabilities, such as rclone.

=Security and Data Management=

'''PIC Credentials:'''
: Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.

Hadoop Distributed File System (HDFS)

2025-05-14T13:14:30Z

Sbogaart: /* How to connect to the service */

= Introduction =

PIC offers a WebDAV service over WebHDFS for seamless file transfer and management. This service is designed for users who need to upload, download, and manage large files on HDFS through a familiar file management interface. The WebDAV protocol allows users to access the HDFS in a way that mimics the experience of working with local file systems, offering the flexibility and ease of use of tools like Finder, File Explorer, and rclone, among others.

WebDAV-compatible clients allow for the management of large datasets and offer multithreaded operations for enhanced performance. This makes the WebDAV service ideal for handling large-scale data uploads and downloads, with support for efficient file transfers.

The service also provides a simple and effective way to interface with HDFS, especially for users who prefer a file-system-like experience for managing their data, rather than relying on more technical methods like command-line tools.

= How to connect to the service =

To connect to the WebDAV service, follow the steps below:

# '''Use a WebDAV-compatible client:'''
#*Finder (macOS)
#*File Explorer (Windows)
#*Linux File System (Linux)
#*rclone (CLI)
#*Cyberduck
#*CrossFTP
#*curl
# '''Mount the WebDAV endpoint:'''
#: The WebDAV server is accessible via the following URL: https://webdav.pic.es/ [https://webdav.pic.es/]
# '''Authenticate with your PIC credentials:'''
#: When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.
# '''Browse, upload, and download files:'''
#: Once connected, you will be able to manage your files in the same way you would with any local file system. You can drag and drop files, create directories, and manage large datasets directly on the HDFS.
**''Tip'': Using a multithreaded client like rclone or Cyberduck will help improve upload and download speeds for large files.

= Usage Guidelines=

* File Uploads:

** The WebDAV service allows you to upload files of any size to HDFS.
** For large datasets, it is recommended to use a multithreaded client, as this will optimize the upload process and make it faster and more efficient.

* File Downloads:

** You can also download files from the HDFS to your local machine.
** Multithreading is supported, allowing you to download large files quickly.

* File Management:

** Files and directories can be created, deleted, or renamed directly within your WebDAV-compatible client.

** You can also move files around within your HDFS storage, making file management easier.

=Best Practices=
'''Optimize large file transfers:'''

: To improve performance, use tools like rclone or Cyberduck for multithreaded file transfers. This helps manage the upload and download of large files or large quantities of files more efficiently.

=Troubleshooting=

'''Cannot Connect to the WebDAV Service:'''
:If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.

'''Slow Uploads or Downloads:'''
:If you experience slow file transfer speeds, check if your client supports multithreaded transfers and consider using a tool like rclone or Cyberduck, which support multiple threads for faster transfers.

'''File Uploads Fail'''
:If file uploads fail, try splitting large files into smaller chunks or use an alternative tool that supports better error handling and retry capabilities, such as rclone.

=Security and Data Management=

'''PIC Credentials:'''
: Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.

Hadoop Distributed File System (HDFS)

2025-05-14T13:13:32Z

Sbogaart: /* Introduction */

= Introduction =

PIC offers a WebDAV service over WebHDFS for seamless file transfer and management. This service is designed for users who need to upload, download, and manage large files on HDFS through a familiar file management interface. The WebDAV protocol allows users to access the HDFS in a way that mimics the experience of working with local file systems, offering the flexibility and ease of use of tools like Finder, File Explorer, and rclone, among others.

WebDAV-compatible clients allow for the management of large datasets and offer multithreaded operations for enhanced performance. This makes the WebDAV service ideal for handling large-scale data uploads and downloads, with support for efficient file transfers.

The service also provides a simple and effective way to interface with HDFS, especially for users who prefer a file-system-like experience for managing their data, rather than relying on more technical methods like command-line tools.

= How to connect to the service =

To connect to the WebDAV service, follow the steps below:

# '''Use a WebDAV-compatible client:'''
#*Finder (macOS)
#*File Explorer (Windows)
#*Linux File System (Linux)
#*rclone (CLI)
#*Cyberduck (macOS, Windows)
#*CrossFTP (macOS, Windows, Linux)
# '''Mount the WebDAV endpoint:'''
#: The WebDAV server is accessible via the following URL: https://webdav.pic.es/ [https://webdav.pic.es/]
# '''Authenticate with your PIC credentials:'''
#: When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.
# '''Browse, upload, and download files:'''
#: Once connected, you will be able to manage your files in the same way you would with any local file system. You can drag and drop files, create directories, and manage large datasets directly on the HDFS.
**''Tip'': Using a multithreaded client like rclone or Cyberduck will help improve upload and download speeds for large files.

= Usage Guidelines=

* File Uploads:

** The WebDAV service allows you to upload files of any size to HDFS.
** For large datasets, it is recommended to use a multithreaded client, as this will optimize the upload process and make it faster and more efficient.

* File Downloads:

** You can also download files from the HDFS to your local machine.
** Multithreading is supported, allowing you to download large files quickly.

* File Management:

** Files and directories can be created, deleted, or renamed directly within your WebDAV-compatible client.

** You can also move files around within your HDFS storage, making file management easier.

=Best Practices=
'''Optimize large file transfers:'''

: To improve performance, use tools like rclone or Cyberduck for multithreaded file transfers. This helps manage the upload and download of large files or large quantities of files more efficiently.

=Troubleshooting=

'''Cannot Connect to the WebDAV Service:'''
:If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.

'''Slow Uploads or Downloads:'''
:If you experience slow file transfer speeds, check if your client supports multithreaded transfers and consider using a tool like rclone or Cyberduck, which support multiple threads for faster transfers.

'''File Uploads Fail'''
:If file uploads fail, try splitting large files into smaller chunks or use an alternative tool that supports better error handling and retry capabilities, such as rclone.

=Security and Data Management=

'''PIC Credentials:'''
: Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.

Hadoop Distributed File System (HDFS)

2025-05-14T12:47:13Z

Sbogaart: The WebDAV service over WebHDFS offered by PIC provides a user-friendly and efficient way to interact with your data on HDFS. Whether you're uploading large files, managing your datasets, or downloading files, this service simplifies the process by integrating with common file management tools.

= Introduction =

PIC offers a WebDAV service over WebHDFS for seamless file transfer and management. This service is designed for users who need to upload, download, and manage large files on HDFS through a familiar file management interface. The WebDAV protocol allows users to access the HDFS in a way that mimics the experience of working with local file systems, offering the flexibility and ease of use of tools like Finder, File Explorer, and rclone, among others.

WebDAV-compatible clients allow for the management of large datasets and offer multithreaded operations for enhanced performance. This makes the WebDAV service ideal for handling large-scale data uploads and downloads, with support for efficient file transfers.

The service also provides a simple and effective way to interface with HDFS, especially for users who prefer a file-system-like experience for managing their data, rather than relying on more technical methods like command-line tools.

== How to connect to the service ==
To connect to the WebDAV service, follow the steps below:

1. Use a WebDAV-compatible client:
Finder (macOS)
File Explorer (Windows)
Linux File System (Linux)
rclone (CLI)
Cyberduck (macOS, Windows)
CrossFTP (macOS, Windows, Linux)

2. Mount the WebDAV endpoint:
The WebDAV server is accessible via the following URL: [https://webdav.pic.es/hdfs]

3. Authenticate with your PIC credentials:
When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.

4. Browse, upload, and download files:
Once connected, you will be able to manage your files in the same way you would with any local file system. You can drag and drop files, create directories, and manage large datasets directly on the HDFS.
Tip: Using a multithreaded client like rclone or Cyberduck will help improve upload and download speeds for large files.

==Usage Guidelines==
- File Uploads:
· The WebDAV service allows you to upload files of any size to HDFS.
· For large datasets, it is recommended to use a multithreaded client, as this will optimize the upload process and make it faster and more efficient.
- File Downloads:
· You can also download files from the HDFS to your local machine.
· Multithreading is supported, allowing you to download large files quickly.
- File Management:
· Files and directories can be created, deleted, or renamed directly within your WebDAV-compatible client.
· You can also move files around within your HDFS storage, making file management easier.

==Best Practices==
1. Optimize large file transfers:
To improve performance, use tools like rclone or Cyberduck for multithreaded file transfers. This helps manage the upload and download of large files or large quantities of files more efficiently.

==Troubleshooting==

1. Cannot Connect to the WebDAV Service:
If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.
2. Slow Uploads or Downloads:
If you experience slow file transfer speeds, check if your client supports multithreaded transfers and consider using a tool like rclone or Cyberduck, which support multiple threads for faster transfers.
3. File Uploads Fail:
If file uploads fail, try splitting large files into smaller chunks or use an alternative tool that supports better error handling and retry capabilities, such as rclone.

==Security and Data Management==

- PIC Credentials:
Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.

Main Page

2025-05-14T12:27:44Z

Sbogaart: /* Services */

== Getting started ==
* [[PIC description|PIC in an image]]
* [[PIC account|Get a PIC account]]
* [[PIC_User_Manual | User manual]]
* [[faq| Frequently asked questions]]

== Services ==
* [[HTCondor]]
* [[Storage]]
* [[JupyterHub]]
* [[Gitlab]]
* [[CosmoHub]]
* [[Hadoop Distributed File System (HDFS)]]
* Spark:
** [[Spark on Hadoop|on Hadoop]]
** [[Spark_on_farm|on HTCondor]]
* [[Transferring data to/from PIC]]

== Experiments ==

* [[Euclid]]
* [[AGN ICE]]
* [[ICFO]]

== More technical information ==

* [[Storage Department]]