Custom Source Flows

These flows show the common ways to run a custom source, from the first sync to deletions, files, permissions, and folders. Each one is a map rather than a full explanation: every step in the list under a diagram links to the section that covers it in detail.

In the diagrams, green is what starts the flow, grey steps call Guru's API, and black steps happen in your own system.

Choose a flow

Every flow starts from a source created with trackStatus: true, as described in Step 1: Create the source. Store the source id and object type ids it returns (Step 2).

Full sync

Push every record on every run. Guru ends up matching your system exactly, and records deleted in your system disappear on their own.

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W1(["Your sync job runs"])
    A1["Open the sync<br/>START_INITIAL or START"]
    B1["Push every current record<br/>with the sync number"]
    C1["Close the sync<br/>with COMPLETE"]
    D1["Guru removes records<br/>you did not push"]
    Z1["Push each record, tagged with its folder id"]
    W1 --> A1 --> B1 --> C1 --> D1

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W1 start
    class A1,B1,C1,D1 guru
    class Z1 spacer

1. Open the sync with START_INITIAL the first time and START after that, and read the sync number from the response.
2. Push each record with that sync number. A record you push again with the same externalId is replaced.
3. Close the sync with COMPLETE. Every record you didn't push is removed. If the run can't finish, close it with FAIL instead, which removes nothing.

Incremental sync with cleanup

Push only what changed since the last successful sync. Records you don't push stay as they are, so deletions need a separate cleanup pass.

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W2(["Your sync job runs"])
    L2["Read lastSyncedDate"]
    Q2["Find records changed<br/>at or after that time"]
    A2["Open the sync and push<br/>each changed record"]
    C2["Close with<br/>COMPLETE_INCREMENTAL"]
    Z2["Push each record, tagged with its folder id"]
    W2 --> L2 --> Q2 --> A2 --> C2

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W2 start
    class L2,A2,C2 guru
    class Q2 external
    class Z2 spacer

And on a schedule, a cleanup pass removes records deleted in your system:

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W3(["Cleanup runs,<br/>for example nightly"])
    A3["Open a full sync<br/>with START"]
    T3["Touch each record that<br/>still exists in your system"]
    C3["Close with COMPLETE"]
    D3["Guru removes records<br/>you did not touch"]
    Z3["Push each record, tagged with its folder id"]
    W3 --> A3 --> T3 --> C3 --> D3

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W3 start
    class A3,T3,C3,D3 guru
    class Z3 spacer

Each run:

1. Read the object type's lastSyncedDate with fieldDetail=FULL. It's the time the last successful sync started.
2. Find the records in your system modified at or after that time.
3. Open the sync with START, push each changed or new record, and close with COMPLETE_INCREMENTAL (see Incremental syncs).

On a schedule, remove records deleted in your system:

1. Open a full sync on the object type with START.
2. Touch every record that still exists in your system. A touch sends no body.
3. Close with COMPLETE. Every record you didn't touch is removed.

On a source whose only object type is RECORD, you can instead delete each record directly when it's deleted in your system.

Push a single change

Mirror one change as it happens, without opening a sync. Useful alongside either sync style when your system can notify you of edits.

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W4(["A record changes<br/>in your system"])
    P4["Push it without<br/>a sync number"]
    G4["Guru creates or<br/>replaces it right away"]
    Z4["Push each record, tagged with its folder id"]
    W4 --> P4 --> G4

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W4 start
    class P4,G4 guru
    class Z4 spacer

1. Push the record and leave out syncNumber. Guru creates or replaces it right away.
2. If you also run full syncs, keep the record in them. A full sync closed with COMPLETE removes any record it didn't push or touch.

File library

Upload files and let Guru extract their text, skipping files that haven't changed since the last run.

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W5(["Your sync job runs"])
    A5["Open the sync"]
    M5["For each file, read<br/>its stored checksum"]
    U5["Touch it if unchanged,<br/>otherwise upload it"]
    C5["Close with COMPLETE"]
    Z5["Push each record, tagged with its folder id"]
    W5 --> A5 --> M5 --> U5 --> C5

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W5 start
    class A5,M5,U5,C5 guru
    class Z5 spacer

1. Create a source with a RECORD object type and any tags you want to filter by (Step 1), then open the sync.
2. For each file, read its stored checksum. If it matches the file's current checksum, touch the record. Otherwise, upload the file with its new checksum and tags.
3. Close the sync with COMPLETE. Files that are gone from your library are removed.

Native permissions

Mirror who can see each record in your system. Permissions ride along with the record sync.

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W6(["Your sync job runs"])
    A6["Open the record sync with<br/>permission dependents"]
    P6["For each record, push it<br/>and set its permissions"]
    G6["Sync groups whose<br/>members changed"]
    C6["Close with COMPLETE<br/>and the same dependents"]
    Z6["Push each record, tagged with its folder id"]
    W6 --> A6 --> P6 --> G6 --> C6

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W6 start
    class A6,P6,G6,C6 guru
    class Z6 spacer

1. Create the source with useSourceNativePermissions: true (Enable native permissions). It can only be set at creation.
2. Open the record sync with USER, GROUP, GROUP_MEMBER, and OBJECT_ACCESS in dependentObjectTypeIds.
3. For each record, push it, then set its complete permission list.
4. Sync any groups whose membership changed.
5. Close the record sync with COMPLETE and the same dependentObjectTypeIds, or the permission syncs stay open.

Folders

Mirror a folder tree so people can filter by folder and scope Knowledge Agents to part of it. The tree has its own sync, which runs before the record sync.

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W7(["Your sync job runs"])
    A7["Open the folder sync<br/>with the tag config id"]
    P7["Push each folder,<br/>in any order"]
    X7["Set folder permissions,<br/>if you use them"]
    C7["Close the folder sync<br/>with COMPLETE"]
    Z7["Push each record, tagged with its folder id"]
    W7 --> A7 --> P7 --> X7 --> C7

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W7 start
    class A7,P7,X7,C7 guru
    class Z7 spacer

Then the record sync:

%%{init: {"theme": "base", "themeVariables": {"fontSize": "16px", "lineColor": "#9C9C9C"}, "flowchart": {"nodeSpacing": 40, "rankSpacing": 45, "padding": 12}}}%%
flowchart TB
    W8(["Folder sync is closed"])
    A8["Open the record sync"]
    P8["Push each record, tagged<br/>with its folder id"]
    C8["Close the record sync"]
    Z8["Push each record, tagged with its folder id"]
    W8 --> A8 --> P8 --> C8

    classDef start fill:#56D886,stroke:#56D886,color:#080B0E
    classDef guru fill:#F0F0F0,stroke:#9C9C9C,color:#080B0E,rx:8,ry:8
    classDef external fill:#080B0E,stroke:#080B0E,color:#FFFFFF,rx:8,ry:8
    classDef spacer fill:none,stroke:none,color:transparent
    class W8 start
    class A8,P8,C8 guru
    class Z8 spacer

1. Declare a HIERARCHICAL tag and a facet over it when you create the source (Declare the folder tag).
2. Open the folder sync with the tag config id in place of an object type id.
3. Push each folder, in any order. If the source uses native permissions, set each folder's permissions after pushing it.
4. Close the folder sync with COMPLETE. Folders you didn't push are removed.
5. Run the record sync, tagging each record with its folder's externalId.
6. To scope a Knowledge Agent to a folder, see Filter by folder.