Skip to content

ENH: improve upload.sh to retry, extract from README.md into a script to automatically test blazegraph #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open

ENH: improve upload.sh to retry, extract from README.md into a script to automatically test blazegraph #16

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d9a2b14
updating nidm files to include collection
dbkeator Jul 24, 2023
03c24d0
updating freesurfer CDEs file to support FS 7.X where some region nam…
dbkeator Sep 28, 2023
02356b8
updating instructions for blazegraph for m1 chip support
dbkeator Jan 12, 2024
fb99342
upload -- provide path directly without cp, but also try up to 5 time…
yarikoptic Feb 9, 2024
7ec48c3
Initial version of the a complete test for blazegraph scripted
yarikoptic Feb 9, 2024
02e7768
RF: split into two scripts so we could run test under another graphdb
yarikoptic Feb 9, 2024
44369b8
Time running of the query, do not show full diff, provide "Accept:" h…
yarikoptic Feb 10, 2024
33dedc6
Make curl silent for upload
yarikoptic Feb 10, 2024
badb6ee
Make upload of query silent
yarikoptic Feb 10, 2024
b9c8f9a
Use GNU parallel to expedite uploads a little -- shaves off about 25%…
yarikoptic Feb 10, 2024
868f6b1
Use variable for API URL in upload.sh
yarikoptic Mar 13, 2024
c7d1d4e
A version of a runner under oxigraph
yarikoptic Mar 13, 2024
b2a1ae2
Add graphDB files
surchs Mar 14, 2024
269f7f8
Some changes to make graphDB happy
surchs Mar 14, 2024
2b7200a
Make upload.sh possibly upload jsonld if supported or ttl depending o…
yarikoptic Mar 15, 2024
f720531
Bring back pipefail
yarikoptic Mar 15, 2024
1a6290e
Merge pull request #1 from surchs/seb_enhs
yarikoptic Mar 15, 2024
957d867
Add QUERY_URL identical to API_URL for blaegraph and oxigraph
yarikoptic Mar 15, 2024
0603b87
Rename _API to _UPLOAD URL for clarity
yarikoptic Mar 15, 2024
7c7d7d2
Fix up URLs for oxi
yarikoptic Mar 18, 2024
d2b9e35
Use 0.4.0-rc.1 of oxigraph not 8 month old "latest"
yarikoptic Sep 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

1. start blazegraph

`docker run --name blazegraph --rm -d -p 8889:8080 lyrasis/blazegraph:2.1.5`
`docker run --name blazegraph --rm -d -p 8889:8080 islandora/blazegraph:3.1.1`

2. to upload all the turtle files, from the root of this repo do:

Expand Down Expand Up @@ -85,4 +85,4 @@ sparql.setQuery(query)
sparql.setReturnFormat(JSON)
results = sparql.query().convert()
print(len(results['results']['bindings']))
```
```
212 changes: 211 additions & 1 deletion cde/fs_cde.ttl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1864,7 +1864,7 @@ fs:fs_000137 a fs:DataElement ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738264> ;
nidm:hasUnit "mm" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0000956> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0111689> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Mean Thickness (mm)" ;
fs:fsunit "mm" ;
fs:isAbout_label "cerebral cortex" ;
Expand Down Expand Up @@ -47962,5 +47962,215 @@ fs:fs_003579 a fs:DataElement ;
fs:structure_id 255 ;
fs:unit "MR" .

fs:fs_003580 a fs:DataElement ;
rdfs:label "Left-Thalamus NVoxels" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0102597> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "voxel" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0112559> ;
fs:description "Left-Thalamus Number of Voxels (unitless)" ;
fs:fsunit "unitless" ;
fs:key_source "Table" ;
fs:measure "NVoxels" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "unitless" .

fs:fs_003581 a fs:DataElement ;
rdfs:label "Left-Thalamus Volume_mm3 (mm^3)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738276> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "mm^3" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0112559> ;
fs:description "Left-Thalamus Volume (mm^3)" ;
fs:fsunit "mm^3" ;
fs:key_source "Table" ;
fs:measure "Volume_mm3" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "mm^3" .

fs:fs_003582 a fs:DataElement ;
rdfs:label "Left-Thalamus normMean (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738264> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Left-Thalamus Intensity normMean (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normMean" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "MR" .

fs:fs_003583 a fs:DataElement ;
rdfs:label "Left-Thalamus normStdDev (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738265> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Left-Thalamus Itensity normStdDev (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normStdDev" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "MR" .

fs:fs_003584 a fs:DataElement ;
rdfs:label "Left-Thalamus normMin (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738266> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Left-Thalamus Intensity normMin (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normMin" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "MR" .

fs:fs_003585 a fs:DataElement ;
rdfs:label "Left-Thalamus normMax (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738267> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Left-Thalamus Intensity normMax (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normMax" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "MR" .

fs:fs_003586 a fs:DataElement ;
rdfs:label "Left-Thalamus normRange (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738268> ;
nidm:hasLaterality "Left" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Left-Thalamus Intensity normRange (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normRange" ;
fs:structure "Left-Thalamus" ;
fs:structure_id 10 ;
fs:unit "MR" .

fs:fs_003587 a fs:DataElement ;
rdfs:label "Right-Thalamus NVoxels" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0102597> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "voxel" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0112559> ;
fs:description "Right-Thalamus Number of Voxels (unitless)" ;
fs:fsunit "unitless" ;
fs:key_source "Table" ;
fs:measure "NVoxels" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "unitless" .

fs:fs_003588 a fs:DataElement ;
rdfs:label "Right-Thalamus Volume_mm3 (mm^3)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738276> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "mm^3" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0112559> ;
fs:description "Right-Thalamus Volume (mm^3)" ;
fs:fsunit "mm^3" ;
fs:key_source "Table" ;
fs:measure "Volume_mm3" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "mm^3" .

fs:fs_003589 a fs:DataElement ;
rdfs:label "Right-Thalamus normMean (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738264> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Right-Thalamus Intensity normMean (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normMean" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "MR" .

fs:fs_003590 a fs:DataElement ;
rdfs:label "Right-Thalamus normStdDev (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738265> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Right-Thalamus Itensity normStdDev (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normStdDev" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "MR" .

fs:fs_003591 a fs:DataElement ;
rdfs:label "Right-Thalamus normMin (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738266> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Right-Thalamus Intensity normMin (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normMin" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "MR" .

fs:fs_003592 a fs:DataElement ;
rdfs:label "Right-Thalamus normMax (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738267> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Right-Thalamus Intensity normMax (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normMax" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "MR" .

fs:fs_003593 a fs:DataElement ;
rdfs:label "Right-Thalamus normRange (MR)" ;
nidm:datumType <http://uri.interlex.org/base/ilx_0738268> ;
nidm:hasLaterality "Right" ;
nidm:hasUnit "https://surfer.nmr.mgh.harvard.edu/MR" ;
nidm:isAbout <http://purl.obolibrary.org/obo/UBERON_0001897> ;
nidm:measureOf <http://uri.interlex.org/base/ilx_0738276> ;
fs:description "Right-Thalamus Intensity normRange (MR)" ;
fs:fsunit "MR" ;
fs:key_source "Table" ;
fs:measure "normRange" ;
fs:structure "Right-Thalamus" ;
fs:structure_id 49 ;
fs:unit "MR" .

fs:DataElement rdfs:subClassOf nidm:DataElement .

49 changes: 49 additions & 0 deletions data-config.ttl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#
# RDF4J configuration template for a GraphDB repository
#
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rep: <http://www.openrdf.org/config/repository#>.
@prefix sr: <http://www.openrdf.org/config/repository/sail#>.
@prefix sail: <http://www.openrdf.org/config/sail#>.
@prefix graphdb: <http://www.ontotext.com/config/graphdb#>.

[] a rep:Repository ;
rep:repositoryID "testplace" ;
rdfs:label "" ;
rep:repositoryImpl [
rep:repositoryType "graphdb:SailRepository" ;
sr:sailImpl [
sail:sailType "graphdb:Sail" ;

graphdb:read-only "false" ;

# Inference and Validation
graphdb:ruleset "rdfsplus-optimized" ;
graphdb:disable-sameAs "true" ;
graphdb:check-for-inconsistencies "false" ;

# Indexing
graphdb:entity-id-size "32" ;
graphdb:enable-context-index "false" ;
graphdb:enablePredicateList "true" ;
graphdb:enable-fts-index "false" ;
graphdb:fts-indexes ("default" "iri") ;
graphdb:fts-string-literals-index "default" ;
graphdb:fts-iris-index "none" ;

# Queries and Updates
graphdb:query-timeout "0" ;
graphdb:throw-QueryEvaluationException-on-timeout "false" ;
graphdb:query-limit-results "0" ;

# Settable in the file but otherwise hidden in the UI and in the RDF4J console
graphdb:base-URL "http://example.org/owlim#" ;
graphdb:defaultNS "" ;
graphdb:imports "" ;
graphdb:repository-type "file-repository" ;
graphdb:storage-folder "storage" ;
graphdb:entity-index-size "10000000" ;
graphdb:in-memory-literal-properties "true" ;
graphdb:enable-literal-index "true" ;
]
].
Loading