GuidesREST API ReferenceRelease Notes
Guides

Automated Backups

Schedule exports of your research data

Bipsync features a tool called the Archiver which supports configurable, scheduled data exports. It ingests a configuration pipeline which allows clients to specify:

  • The types of data which should be exported
  • Any criteria which should be used to filter the exported documents
  • The file type for each exported document
  • The folder structure under which the documents will be stored
  • The delivery destination

Backups can be either scheduled to run at certain times of the day, or they can be requested on-demand via Bipsync's API. Scheduled backups are more flexible and must be configured by a Bipsync CSM, but on-demand backups can be requested by clients as required.

Exportable Data Types

All of the research data held within Bipsync is eligible for export. A "full" backup is comprised of:

  • Research (notes and files)
  • Entities (this will vary depending on each clients' schema but can include security masters, portfolios, investments, ideas, etc.)
  • Contacts
  • Comments

Each data type can be given its own configuration, which then dictates exactly how that data type is exported. Clients can also elect to export a subset of data types — just contacts, for example, or all research documents with their comments.

Export Criteria

Each data type can be given criteria to limit the documents which are included in the backup. There isn't really any limit on the criteria which can be provided, but some common examples include:

  • Records since a given date
  • Records created by a given author or authors
  • Records tagged with a given entity

If there's a particular set of documents which are to be exported, there's almost always a way to identify and include them. Your Bipsync CSM can advise you as to best approach.

File Types

Each record in the export is placed within a file. In the case of research notes the file type can be specified.

The file types that Bipsync supports are:

  • JSON
  • PDF
  • Word
  • HTML

JSON is the good format to use if you plan on ingesting the data into a system programatically. Otherwise PDF, Word and HTML are good options if the research documents are to be consumed by a human.

Folder Structure

The files which are created for each record are stored in a folder structure before being zipped up into an archive. The folder structure is configurable, and can reference each documents' metadata to determine where the file should be stored.

For example, research can be stored under a path which references the note type, the author, and the month in which the document was created — an example may look like this:

/research/meeting-notes/jane-doe/2019/Oct/10-10-2019-monday-morning-meeting-note.docx

Again, a Bipsync CSM can help craft the folder structure to your requirements.

Delivery

Once the archive has been prepared, it can be delivered to one of the following locations:

  • An S3 Bucket within the clients' Amazon Web Services account
  • A link which is sent to a specified email address
  • An SFTP Server

Clients can choose their preferred destination. If the backup is being prepared on a schedule, it will arrive at the chosen location a short time after the allotted time once the archive has been created, compressed and uploaded.

On-Demand Backups

Clients are able to request their own backups on-demand via Bipsync's API.

See this section of our API documentation to learn about the endpoints you'll need to use, but here's a brief overview:

A POST HTTP request is sent to /v2/archiver/requestArchive. This will trigger an export using the pre-configured backup configuration as a starting point. Additional parameters can be provided to further configure the export, such as:

  • since to limit the exported documents to only those changed since the given date
  • deliver to specify the delivery method
  • generatePdfs to also archive notes as PDFs
  • generateHtml to also archive notes as HTML
  • generateWord to also archive notes as Word documents

That endpoint will return an export job ID. This ID can be incorporated into this API URL to check on the progress of the export:/v2/archiver/job/{job}.

Differential Backups

Differential backups contain the data that has changed since the last backup was taken. They offer some advantages over "full" backups:

  • They are quicker to produce and transmit
  • They are quicker to download and decompress
  • They use fewer resources and so are more environmentally friendly

They're easily produced either though configuration by Bipsync, or by passing the since parameter when calling the on-demand backup API URL. Just provide the date and time of the last successful backup, and the subsequent export will contain everything that's new in Bipsync since that time.

Updated 7 months ago