Certain urls in XNAT can be very expensive to support on larger instances (> 1,000,000 rows). Urls like /data/experiments and /data/subjects will return every experiment or subject that a user has access to. For users with access to a lot of data, this call can get quite expensive and cause performance issues in the server. Furthermore, a table of a million subjects is problematic to make use of in most environments.
When using urls which return a table of experiments or subjects, it is highly suggested that developers (of plugins and scripts) make use of paging to better manage their results. Starting in XNAT 1.8.9, parameters (limit & offset) were added to allow users to manage the number of rows being returned by a user.
Page 1 - GET - /data/experiments?limit=100&offset=0
Page 2 - GET - /data/experiments?limit=100&offset=100
Page 3 - GET - /data/experiments?limit=100&offset=200
Page 4 - GET - /data/experiments?limit=100&offset=300
Making use of these parameters will prevent the server from being unnecessarily stressed and keep your local data in memory at a more usable amount.
By default, XNAT returns ALL data a user has access to (for backwards compatibility). In future versions, administrators may be able to change that default behavior to instead default to paged results. This would likely be welcomed on some of the largest servers where those urls are pretty unusable when not paged. If you are coding against these urls and want to make sure you get ALL results, use 'limit=*' to return all results (in case the default behavior changes in future versions).
When results are paged, the 'totalRecords' variable is not included in the returned results (as calculating it would require a potentially expensive query).