{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":[]},"type":"markdown"},"seo":{"title":"Data Sources","description":"Hyperproof developer resources for custom integrations.","llmstxt":{"hide":false,"sections":[{"title":"Table of contents","includeFiles":["**/*"],"excludeFiles":[]}],"excludeFiles":[]}},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"data-sources","__idx":0},"children":["Data Sources"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A data source is a Hypersync app component that retrieves data from an external service. Data sources can access external services in a variety of ways including REST, GraphQL, and direct database access. Each Hypersync app should contain one data source component that implements the IDataSource interface or derives from a base class that implements IDataSource."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Data sources can retrieve different data sets from the external service. Each data set is accessed by name. For example, a Jira data source might expose three data sets: one for issues, one for labels, and one for users."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Data set names are used in other components like proof types to identify the specific data element or elements needed by that component."]},{"$$mdtype":"Tag","name":"blockquote","attributes":{},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["NOTE"]},": Data sets can and ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["should"]}," be re-used across multiple proof types, and can also be used in non-proof scenarios such as collecting user data during the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["validateCredentials"]}," process."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"rest-data-sources","__idx":1},"children":["REST Data Sources"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For services that expose their data through a REST API, developers are recommended to derive a data source from the RestDataSourceBase base class. This base class makes it possible to configure the data sets along with filters, sorts, and transformations in a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSources.json"]}," file that is included in your package. You can configure most of your data retrieval functionality without writing any code."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If your external service uses OAuth for authorization (see ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/hypersync-sdk/doc/004-connections"},"children":["Connections"]},"), your app's data source should look like this:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"export class MyServiceDataSource extends RestDataSourceBase {\n  constructor(accessToken: string) {\n    super(config as IRestDataSourceConfig, Messages, {\n      Authorization: `Bearer ${accessToken}`,\n      'Content-Type': 'application/json'\n    });\n  }\n}\n\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You should also have ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["createDataSource"]}," method like this in your app's ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["HypersyncApp"]}," class:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"public async createDataSource(accessToken: string): Promise<IDataSource> {\n  return new MyServiceDataSource(accessToken);\n}\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For services that use some other form of authorization, use the following pattern:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"export class MyServiceDataSource extends RestDataSourceBase {\n  constructor(credentials: CustomAuthCredentials) {\n    // TODO: Update the headers below for your REST service.\n    super(config as IRestDataSourceConfig, Messages, {\n      Authorization: `Basic ...`,\n      'Content-Type': 'application/json'\n    });\n  }\n}\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["And in your HypersyncApp:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"public async createDataSource(credentials: CustomAuthCredentials): Promise<IDataSource> {\n  return new MyServiceDataSource(credentials);\n}\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"datasourcejson-file","__idx":2},"children":["dataSource.json File"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Once you have created your app's ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["RestDataSourceBase"]}," component and updated the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["createDataSource"]}," method in your ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["HypersyncApp"]},", add dataSource.json file under the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["/json"]}," directory. The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["RestDataSourceBase"]}," base class will automatically load this configuration file when it is instantiated."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For more information on configuring the data sets in your data source using ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSource.json"]},", see the ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/hypersync-sdk/doc/052-data-source-json"},"children":["dataSource.json Format page"]},"."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"paging","__idx":3},"children":["Paging"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Many REST APIs use a paging mechanism to allow data to be retrieved in chunks. For example, some APIs take a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageSize"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageNumber"]}," argument which specify how many items to return, and the page number to start reading from, respectively."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Four paging styles are supported: ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Page Based"]},", ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Offset And Limit"]},", ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Next Token"]},", and ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["GraphQL Connections"]},". As a default, query string parameters will be programmatically added to an API url. If POST is designated as the data source HTTP method, paging parameters are added to the body of the request."]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Page Based."]}," Begin paging at a starting value and increment the page value by 1 after each iteration (1, 2, 3, etc). Return at most ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["limitValue"]}," items per page."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"\"pagingScheme\": {\n  \"type\": \"pageBased\",\n  \"request\": {\n    \"pageParameter\": \"pageNumber\",\n    \"pageStartingValue\": 1,\n    \"limitParameter\": \"pageSize\",\n    \"limitValue\": 100\n  },\n  \"pageUntil\": \"noDataLeft\"\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The mandatory ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["request"]}," property in the paging scheme constructs the paged query string. The query string of the first API call from the above example will be: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["?pageNumber=1&pageSize=100"]},". Each paging scheme must include a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageUntil"]}," property which defines the point at which pagination stops. If ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reachTotalCount"]}," condition is applied, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["totalCount"]}," must be defined in the response object, which represents the path to the total combined number of items in the data returned from the external service.*"]},{"$$mdtype":"Tag","name":"ol","attributes":{"start":2},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Offset And Limit."]}," Begin paging at a starting value and increment the offset by the number of elements in a full page (0, 100, 200, 300, etc). Return at most ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["limitValue"]}," items per page."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"\"pagingScheme\": {\n  \"type\": \"offsetAndLimit\",\n  \"request\": {\n    \"offsetParameter\": \"offset\",\n    \"offsetStartingValue\": 0,\n    \"limitParameter\": \"limit\",\n    \"limitValue\": 100\n  },\n  \"response\": {\n    \"totalCount\": \"pagination.total\"\n  },\n  \"pageUntil\": \"reachTotalCount\"\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The mandatory ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["request"]}," property in the paging scheme constructs the paged query string. The query string of the first API call from the above example will be: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["?offset=0&limit=100"]},". Each paging scheme must include a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageUntil"]}," property which defines the point at which pagination stops. If ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["reachTotalCount"]}," condition is applied, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["totalCount"]}," must be defined in the response object. This string value represents the path to the total combined number of items in the data returned from the external service.*"]},{"$$mdtype":"Tag","name":"ol","attributes":{"start":3},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Next Token."]}," Begin paging and continue until ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["nextToken"]}," is no longer provided. Return at most ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["limitValue"]}," items per page. Tokens may be a unique string returned from the external service or a url."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"\"pagingScheme\": {\n  \"type\": \"nextToken\",\n  \"request\": {\n    \"tokenParameter\": \"token\",\n    \"limitParameter\": \"size\",\n    \"limitValue\": 20\n  },\n  \"response\": {\n    \"nextToken\": \"next.token\"\n  },\n  \"pageUntil\": \"noNextToken\",\n  \"tokenType\": \"token\"\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The mandatory ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["request"]}," property in the paging scheme constructs the paged query string. The query string of the first API call from the above example will be: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["?size=20"]},". Each successive call will be structured in the pattern: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["?size=20&token=891b629672384d04"]},". Each paging scheme must include a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageUntil"]}," property which defines the point at which pagination stops. When ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["noNextToken"]}," condition is applied, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["nextToken"]}," must be included in the response object. This string value represents the path to the expected value in the data returned from the external service.*"]},{"$$mdtype":"Tag","name":"ol","attributes":{"start":4},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["GraphQL Connections."]}," Following the GraphQL ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"https://graphql.org/learn/pagination/#connection-specification"},"children":["Connections"]}," specification, continue paging until ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["hasNextPage"]}," is false. Return at most ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["limitValue"]}," items per page. Supports forward, non-nested pagination."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"\"body\": {\n  \"query\": \"query($first: Int, $after: String) { attributes(first: $first, after: $after) { nodes { id name } pageInfo { endCursor hasNextPage } } }\",\n  \"variables\": {\n    \"first\": 500\n  }\n},\n\"method\": \"POST\",\n\"property\": \"data.attributes.nodes\",\n\"pagingScheme\": {\n  \"type\": \"graphqlConnections\",\n  \"request\": {\n    \"limitParameter\": \"first\",\n    \"limitValue\": 500\n  },\n  \"response\": {\n    \"pageInfo\": \"data.attributes.pageInfo\"\n  },\n  \"pageUntil\": \"noNextPage\"\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The paging scheme dynamically adds the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["first"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["after"]}," variables to the body of a request. The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["after"]}," variable is defined using the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["endCursor"]}," string from the preceding response. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageInfo"]}," must be included in the paging scheme response object. This string value represents the path to the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["pageInfo"]}," object in the data returned from the external service."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["*If values are to be found in the response header, apply the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["header:"]}," prefix."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"custom-data-sources","__idx":4},"children":["Custom Data Sources"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For services that do not expose data as REST, or for services that use certain REST patterns that are incompatible with RestDataSourceBase, the Hypersync SDK makes it possible to create a custom data source."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To begin, define your data source class using one of the patterns below."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["OAuth Authorization"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"export class MyServiceDataSource implements IDataSource {\n  private accessToken: string;\n\n  constructor(accessToken: string) {\n    this.accessToken = accessToken;\n  }\n}\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Custom Authentication"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"export class MyServiceDataSource implements IDataSource {\n  private credentials: CustomAuthCredentials;\n\n  constructor(credentials: CustomAuthCredentials) {\n    this.credentials = credentials;\n  }\n}\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Once you have created the class and a properly formatted constructor, all that is left is to implement the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["getData"]}," method. This is the only method defined in the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["IDataSource"]}," interface."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"async getData(\n  dataSetName: string,\n  params?: DataValueMap\n): Promise<DataSetResult<DataObject | DataObject[]>> {\n  // TODO: Retrieve the data set by name using the provided parameters.\n}\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"br","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"br","attributes":{},"children":[]}]}]},"headings":[{"value":"Data Sources","id":"data-sources","depth":1},{"value":"REST Data Sources","id":"rest-data-sources","depth":2},{"value":"dataSource.json File","id":"datasourcejson-file","depth":3},{"value":"Paging","id":"paging","depth":3},{"value":"Custom Data Sources","id":"custom-data-sources","depth":2}],"frontmatter":{"seo":{"title":"Data Sources"}},"lastModified":"2026-04-28T18:11:36.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/hypersync-sdk/doc/005-data-sources","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}