nappMATLAB-toolbox/openBIS/README.md

nappMATLAB → openBIS Sync

Syncs beamtime metadata from a MATLAB FileList table into openBIS as ELN-LIMS entries — including raw file attachments and embedded plot/photo/ video thumbnails.

Requires the MATLAB Napp Toolbox (nappFileList, nappGenerateSpectra, nappPlot, etc.) to already be set up and to have produced a populated FileList table, with campaign metadata set via FileList.Properties.CustomProperties. This repo picks up from there — it doesn't cover importing raw beamtime files, generating spectra, HDF5 export, or the HTML report; those are generic toolbox functionality, not specific to openBIS.

Files

File	Purpose
nappSyncToOpenbis.py	Does the actual openBIS sync via pybis: matching/creating/updating objects, setting properties, linking a parent, uploading raw files, embedding thumbnails. Called from MATLAB — never run standalone.
callOpenbisSync.m	The script you actually run: connects, optionally filters rows, generates plot/photo/video thumbnails, and calls into nappSyncToOpenbis.py to sync.
testpybis.m	Sandbox for testing connection/logic changes on a small row slice before running callOpenbisSync.m on real data. Also holds one-off utilities (e.g. adding a vocabulary term).

callOpenbisSync.m and testpybis.m both depend on nappSyncToOpenbis.py being importable — they're two different entry points into the same engine, not alternatives to each other.

Requirements

• MATLAB (R2025b or later; R2025b supports Python 3.9–3.12 only — Python 3.13 needs R2026a or later. Check MathWorks' compatibility table before choosing a Python version.)
• A Python environment separate from your system/Xcode Python, with pybis installed. A dedicated venv is strongly recommended — installing packages into a Homebrew-managed Python directly will hit macOS's "externally managed environment" guard (PEP 668).
• An openBIS instance with:
  • A Space, and (optionally) a parent object representing the beamtime as a whole, that per-file entries link to as a parent.
  • An object type matching (or adapted from) the property mappings in nappSyncToOpenbis.py's configuration section (see below).
  • A RAW_DATA-type dataset type for attaching original files.

openBIS object type setup

This code does not create an object type for you — an admin needs to define one (e.g. an "Experimental Step" type) in openBIS first, with a property for every entry referenced in nappSyncToOpenbis.py's configuration section (DIRECT_MAP, NUMERIC_MAP, TIMESTAMP_MAP, TEXT_JOIN_MAP, CONSTANT_MAP, CAMPAIGN_MAP, plus FILENUMBER_PROPERTY, CATEGORY_PROPERTY, PHOTON_ENERGY_PROPERTY, KE_RANGE_PROPERTY, RESULTS_PROPERTY), and each property's data type must match what the script expects to write: REAL/INTEGER for anything in NUMERIC_MAP (per its "float"/"int" tag), TIMESTAMP for TIMESTAMP_MAP entries, CONTROLLEDVOCABULARY (with matching terms, e.g. XPS/NEXAFS/PHOTO) for the category property, and VARCHAR/MULTILINE_VARCHAR for everything else. A mismatch here isn't silent — openBIS returns a clear validation error naming the property and the type it expected — but it's still much faster to get the schema right up front than to debug it property-by-property. get_property_assignments() (already in nappSyncToOpenbis.py) is a read-only way to dump an existing type's actual current structure, useful as ground truth when setting up a new one or checking one for drift.

Setup

python3 -m venv ~/matlab-pybis-venv
source ~/matlab-pybis-venv/bin/activate
pip install --upgrade pip
pip install pybis

In MATLAB, every session:

pyenv('Version', '/path/to/matlab-pybis-venv/bin/python3')
py.sys.path().append('/path/to/folder/containing/nappSyncToOpenbis.py');
py.importlib.import_module('nappSyncToOpenbis');

pyenv and the Python module import don't persist across a MATLAB restart — both need re-running at the start of every session.

Authentication

Personal Access Tokens (PATs) can be created and listed via the API, but PAT-based authentication may be disabled at the server-config level independent of that — this is a system-admin setting (confirmed by openBIS developers), not something fixable from a script. If PATs fail with "user is not logged in" despite a valid, unexpired token, ask your openBIS system admin whether PAT authentication is enabled for your instance. Username/password (o.login(...)) is unaffected by this and works regardless.

Usage

With a populated FileList already in your MATLAB workspace (via the Napp Toolbox): fill in connection details at the top of callOpenbisSync.m (URL, space, object type, parent identifier), set rowFilter if you only want a subset, and run it. Test changes on a small slice via testpybis.m first, rather than editing nappSyncToOpenbis.py and running the full sync straight away.

Known limitations

• Video thumbnails always use the first frame of the file — not necessarily the most representative one. Change the frame read in callOpenbisSync.m's thumbnail loop if that matters for your data.
• Re-running the sync on a row that already has embedded images in its Results field will append the images again (no de-duplication).
• VideoReader's format support depends on codecs available on the machine running MATLAB.

pybis gotchas this code works around

Found by reading pybis 1.37.5's own source directly rather than guessing — worth checking whether these still apply if you upgrade pybis:

• Property codes are lowercase at the wire level, even though the admin UI and xlsx schema exports display them uppercase.
• get_parents() is buggy: it populates the internal parent cache as a flat list of plain identifier strings, a different shape than add_parents()/set_parents() produce. Merely calling it corrupts state that save() later can't handle — use add_parents() directly, without a preceding get_parents() check.
• Iterating a Things collection can be buggy (e.g. from get_terms()) — it internally calls a single-item fetch method missing a required argument. Read .df (the underlying DataFrame) directly instead.
• set_token()/is_session_active() both call the isSessionActive RPC, which can reject a token that's genuinely valid per other endpoints (see Authentication above). Writing directly to o.__dict__["token"] bypasses the check without affecting anything else.
• Passing collection=None explicitly (rather than omitting the kwarg) causes a malformed "experimentId": {} in the create request, which the server rejects. Only include collection in the kwargs dict when it has a real value.
• Values crossing from MATLAB via pyargs(...) (not through jsonencode/json.loads) don't always arrive as genuine native Python types — a bare string can end up wrapped in a way that breaks pybis's own internal type checks, and MATLAB's [] can cross as a NumPy array whose truthiness is ambiguous. Anything simple crossing this way gets coerced explicitly (see _clean_str_or_none in nappSyncToOpenbis.py).
• jsonencode(table2struct(T)) for a single-row table produces a JSON object, not a one-element array (MATLAB's table2struct gives a scalar struct, not a 1-element struct array, for one row). sync_rows normalizes this on the way in, but it's worth knowing if you ever build the JSON payload a different way.

Adapting for a different openBIS object type

nappSyncToOpenbis.py is split into a CONFIGURATION section (plain dicts: column → property mappings, fixed values, text-join rules) and an ENGINE section (generic logic that reads that config). To point this at a different object type, replace the configuration section only — the engine, including all the pybis workarounds above, should keep working unmodified.