Sbogaart: Created page with "= Introduction = PIC provides access to the distributed file system HDFS through a '''VOSpace server''', following the IVOA standard described in [https://www.ivoa.net/docume..."

2025-07-23T11:30:50Z

Created page with "= Introduction = PIC provides access to the distributed file system HDFS through a '''VOSpace server''', following the IVOA standard described in [https://www.ivoa.net/docume..."

New page

= Introduction =

PIC provides access to the distributed file system HDFS through a '''VOSpace server''', following the IVOA standard described in [https://www.ivoa.net/documents/VOSpace/20180502/PR-VOSpace-2.1-20180502.html VOSpace 2.1].

This service is an alternative to WebDAV access, allowing users to manage their data programmatically and in a structured way using tools compatible with the Virtual Observatory (VO) ecosystem. It is especially aimed at users who require data management operations such as reading, writing, moving, and metadata querying within a standardized environment.

= How to connect to the service =

== VOSpace Endpoint ==

The VOSpace server is available at the following URL:

https://vospace.pic.es/vospace

== Compatible clients ==

You can access the service using tools compatible with the VOSpace 2.1 standard:

* [https://pypi.org/project/vos/ '''vos''' client by CADC]
: Install with pip:
pip install vos>=3.6.3

* '''curl''':
: Can be used to perform HTTP operations following the examples defined in the [https://www.ivoa.net/documents/VOSpace/20180502/PR-VOSpace-2.1-20180502.html VOSpace 2.1 standard], including node creation, file transfers, and property queries.

== Authentication ==

The VOSpace server allows both anonymous and authenticated access.
To access restricted data or personal space, users must specify their '''PIC username and password'''.

= Usage Guidelines =

Once connected to the service, users can perform a variety of operations defined by the VOSpace standard. Supported functionalities include:

* '''getProtocols''': Query supported transfer protocols.
* '''getViews''': View available data formats (views).
* '''getProperties''': Obtain node properties.
* '''createNode''': Create files or folders.
* '''getNode''': Retrieve node information.
* '''deleteNode''': Delete files or folders.
* '''moveNode''': Move nodes inside the VOSpace.
* '''put / get''': Upload or download files.
* '''pushToVoSpace / pullFromVoSpace''': Transfer files via external URLs.

= Best Practices =

* Avoid using the '''!''' character in file or folder names, as it may cause compatibility issues with the CADC '''vos''' client.
* Use tools that comply with the VOSpace standard to ensure compatibility and avoid transfer errors.

= Troubleshooting =

'''Cannot access the server:'''
* Verify the URL is correct: <code>https://vospace.pic.es/vospace</code>
* Make sure PIC credentials are set correctly.
* Check client configuration (e.g., '''vos''', '''curl''').

'''Problems with properties:'''
* The '''setProperties''' operation is '''not allowed''' due to permission restrictions on HDFS.

= Security and Data Management =

* Protect PIC credentials. Do not share or store them in plain text.
* For automation, consider using secure credential managers or temporary storage.

= Usage Examples =

== Using the CADC '''vos''' client (Python) ==

This server supports [https://pypi.org/project/vos/ the CADC vos client] and operations like '''listdir''', '''mkdir''', '''copy''', '''move''', and '''delete'''. Minimum version: '''vos >= 3.6.3'''.

=== Configuration ===

* Create a config file (e.g. <code>vos-config.ini</code>):
[vos]
resourceID = https://vospace.pic.es/vospace http

* Export the file path as an environment variable:

export VOSPACE_CONFIG_FILE=/path/to/vos-config.ini

* Create <code>~/.netrc</code> for authentication:

machine https://vospace.pic.es/vospace
login USERNAME
password PASSWORD

* Set permissions:

chmod 600 ~/.netrc
=== Python usage examples ===

import vos

client = vos.Client()

# List contents
files = client.listdir("https://vospace.pic.es/vospace/nodes/user/my_user/")
print("Contents:", files)

# Create directory
client.mkdir("https://vospace.pic.es/vospace/nodes/user/my_user/mydir")

# Upload file
client.copy("localfile.txt", "https://vospace.pic.es/vospace/user/my_user/mydir/localfile.txt")

# Download file
client.copy("https://vospace.pic.es/vospace/user/my_user/mydir/localfile.txt", "downloaded.txt")

# Move file
client.move(
"https://vospace.pic.es/vospace/user/my_user/mydir/localfile.txt",
"https://vospace.pic.es/vospace/user/my_user/mydir/renamedfile.txt"
)

# Get node info
node = client.get_node("https://vospace.pic.es/vospace/nodes/user/my_user/mydir/renamedfile.txt")
print("Properties:", node.props)

# Delete resource
client.delete("https://vospace.pic.es/vospace/nodes/user/my_user/mydir")

'''Note''': Ensure configuration is properly set.

== Using curl ==

You can also interact with the server using '''curl''' or other HTTP tools.

These must follow the [https://www.ivoa.net/documents/VOSpace/20180502/PR-VOSpace-2.1-20180502.html VOSpace 2.1] specification.

'''Important:'''
* Avoid using '''!''' in URLs.
* Use full HTTP paths like <code>/vospace/nodes/...</code>.

= VOSpace Server Endpoints =

The server provides the following REST endpoints per the VOSpace 2.1 specification:

{| class="wikitable"
! Method !! Path !! Description
|-
| GET || <code>/vospace/protocols</code> || Retrieves supported transfer protocols
|-
| GET || <code>/vospace/views</code> || Retrieves available data views
|-
| GET || <code>/vospace/properties</code> || Retrieves node properties
|-
| GET || <code>/vospace/capabilities</code> || Retrieves server capabilities
|-
| GET || <code>/vospace/{job_id}</code> || Retrieves job information
|-
| GET || <code>/vospace/{job_id}/phase</code> || Retrieves job phase
|-
| GET || <code>/vospace/{job_id}/error</code> || Retrieves job error info
|-
| GET || <code>/vospace/{job_id}/results/transferDetails</code> || Retrieves transfer details
|-
| GET || <code>/vospace/{path}</code> || Retrieves a file/folder node
|-
| GET || <code>/vospace/{path}</code> || Downloads a file (streaming)
|-
| PUT || <code>/vospace/{path}</code> || Creates a file or folder
|-
| PUT || <code>/vospace/{path}</code> || Uploads a file
|-
| POST || <code>/vospace/</code> || Moves or copies a node (returns 303 redirect)
|-
| POST || <code>/vospace/synctrans</code> || Push/pull transfer
|-
| POST || <code>/vospace/{path}</code> || Set properties (''not supported'')
|-
| POST || <code>/vospace/{job_id}/phase</code> || Change job phase
|-
| DELETE || <code>/vospace/{path}</code> || Deletes a file or folder
|}

== Notes ==
* <code>{path}</code> and <code>{job_id}</code> are dynamic parameters.
* The server uses standard HTTP error codes: 400, 403, 404, 409, 500.
* '''PUT''' supports uploads and node creation.
* '''POST /vospace/''' moves or copies nodes (responds with '''303 See Other''').
* '''setProperties''' is disabled due to HDFS permissions.
* File downloads return streamed content with '''GET'''.
* Avoid using '''!''' in paths for full compatibility with the CADC client.

HDFS Access via VOSpace - Revision history

Sbogaart: Created page with "= Introduction = PIC provides access to the distributed file system HDFS through a '''VOSpace server''', following the IVOA standard described in [https://www.ivoa.net/docume..."