Skip to content

Commit 064317c

Browse files
committed
feat(search): add GitHub installation indexing
1 parent a89b158 commit 064317c

55 files changed

Lines changed: 4453 additions & 113 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/docs/content/docs/search/connect-your-account.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ For a source configured inside a workspace, join that workspace and use its **Se
5656
| Source setup | Your next step |
5757
| --- | --- |
5858
| Member accounts | Connect your own account, including when you are the admin. |
59+
| GitHub App installation | Connect GitHub once for this Sim organization. The App handles indexing; your account establishes which repositories you may search. |
5960
| Confluence admin/service account | Connect Confluence to verify your identity; the administrator's account handles the crawl. |
6061
| Google Drive delegated service account | No personal connection is needed for that source. Your verified Sim email is matched to Drive permissions. |
6162
| GitLab instance administrator | No personal connection is needed. Your verified Sim email must match a confirmed GitLab email. |

apps/docs/content/docs/search/github.mdx

Lines changed: 64 additions & 16 deletions
Large diffs are not rendered by default.

apps/docs/content/docs/search/index.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,17 +45,18 @@ Source availability depends on the deployment and organization policy. An unavai
4545

4646
## Choose the right connection method
4747

48-
Most sources use member accounts. Google Drive and Confluence also support a central administrator connection; GitLab requires an administrator token for a self-managed instance.
48+
Most sources use member accounts. GitHub supports a central App installation with a personal connection for each reader. Google Drive and Confluence also support a central administrator connection; GitLab requires an administrator token for a self-managed instance.
4949

5050
| Method | What the admin does | What teammates do |
5151
| --- | --- | --- |
5252
| **Member accounts** | Sets the source's filters once. | Connect their own accounts. Sim lists documents using each member's access. |
53+
| **GitHub App installation** | Installs the App, selects it under **Sync documents with**, and adds repository sources. | Connect GitHub once. Sim checks each reader's current repository access before returning installation-indexed content. |
5354
| **Service account** (Drive) / **Admin or service account** (Confluence) | Connects an account that can read the content and the source's permissions or directory. | Join the organization with a matching verified identity. Confluence also requires each person to connect their account. |
5455
| **Administrator token** (GitLab) | Connects a self-managed instance administrator token and selects projects to index. | Join the organization with a verified Sim email matching GitLab. No personal connection is needed. |
5556

5657
Adding a Google Drive or Confluence source from the admin page starts central setup. For personal connections, use **Integrations → Connect account** in the main sidebar. An approved provider can create its first member source there; required repository, site, or project fields are collected before authorization. Admins can edit that source's filters afterward in its **Settings** tab.
5758

58-
Some member sources offer **More options → Sync documents with**. **Connected members** uses members' accounts for both content and access checks. Selecting a dedicated account uses it to fetch content; members still connect to establish which documents they may search. **Account for browsing** only helps an admin pick source options—it does not enroll that account for Search.
59+
Some member sources offer **Sync documents with**, either directly in setup or under **More options**. **Connected members** uses members' accounts for both content and access checks. Selecting a dedicated account uses it to fetch content; members still connect to establish which documents they may search. For GitHub organization sources, choose **Connect GitHub App** in this field to [connect an installation](/search/github#add-a-repository). **Account for browsing** only helps an admin pick source options—it does not enroll that account for Search.
5960

6061
<Callout type="info">
6162
An administrator connection does not grant everyone access to everything. Search applies the source's supported permission rules. It also does not automatically discover every employee's data: the indexing account must be able to read the configured content.
@@ -66,7 +67,7 @@ Some member sources offer **More options → Sync documents with**. **Connected
6667
| Source | Content | Connection in Search |
6768
| --- | --- | --- |
6869
| [Confluence](/search/confluence) | Pages and blog posts | Admin/service account or member accounts; each teammate connects |
69-
| [GitHub](/search/github) | Repository text files | GitHub App installation plus each member's authorization |
70+
| [GitHub](/search/github) | Repository text files | App installation or member indexing; each teammate connects |
7071
| [GitLab](/search/gitlab) | Repository files, wikis, issues, merge requests | Self-managed instance administrator token; no member connection |
7172
| [Gmail](/search/gmail) | Email thread text | Each member's Gmail account |
7273
| [Google Calendar](/search/google-calendar) | Calendar events | Each member's Google Calendar account |
@@ -133,7 +134,7 @@ Workspace Search remains separate. Workspace admins add sources through **Search
133134
3. Ask a teammate with different source access to repeat the search. Documents restricted to you should not appear for them.
134135
4. Change or remove a test document's access in the source and check again after the next completed content and permission refresh.
135136

136-
Search runs background syncs on an hourly schedule. Large sources, provider limits, and indexing queues can delay completion. Results are indexed copies, so edits and access changes are not fetched live for every query.
137+
Search runs background syncs on an hourly schedule. Large sources, provider limits, and indexing queues can delay completion. Results are indexed copies, so edits appear after syncing. Permission refresh behavior depends on the connector: GitHub sources indexed through an App installation also check the reader's current repository access before returning indexed content.
137138

138139
## If indexing needs attention
139140

apps/sim/.env.example

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,17 @@ CRON_SECRET=your_cron_secret # Use `openssl rand -hex 32` to generate. Authentic
143143
# S3_ENDPOINT= # Custom endpoint for S3-compatible storage (Cloudflare R2, MinIO, Backblaze B2). Leave unset for AWS S3
144144
# S3_FORCE_PATH_STYLE=true # Required for MinIO/Ceph RGW. Leave unset for AWS S3 and R2
145145

146+
# GitHub Search (Optional - credentials from a GitHub App with expiring user tokens)
147+
# Use a separate app for production and staging. Allow installation by any account for a public app.
148+
# Grant repository Contents: read and Metadata: read, plus user Email addresses: read.
149+
# Callback: <NEXT_PUBLIC_APP_URL>/api/auth/oauth2/callback/github-repositories
150+
# GITHUB_APP_CLIENT_ID= # GitHub App client ID; distinct from sign-in OAuth credentials
151+
# GITHUB_APP_CLIENT_SECRET=
152+
# Optional organization indexing through an app installation; readers still connect their own GitHub account.
153+
# GITHUB_APP_ID= # Numeric GitHub App ID
154+
# GITHUB_APP_SLUG= # App slug from https://github.com/apps/<slug>
155+
# GITHUB_APP_PRIVATE_KEY= # RSA PEM private key; literal \\n sequences are accepted
156+
146157
# Instagram OAuth (Optional - Instagram App ID/Secret from Meta App Dashboard > Instagram > API setup with Instagram login)
147158
# INSTAGRAM_CLIENT_ID=
148159
# INSTAGRAM_CLIENT_SECRET=
Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
/**
2+
* @vitest-environment node
3+
*/
4+
import { authMockFns } from '@sim/testing'
5+
import { NextRequest, NextResponse } from 'next/server'
6+
import { beforeEach, describe, expect, it, vi } from 'vitest'
7+
8+
const mocks = vi.hoisted(() => ({ list: vi.fn(), connect: vi.fn(), rateLimit: vi.fn() }))
9+
10+
vi.mock('@/lib/core/rate-limiter', () => ({
11+
enforceUserRateLimit: mocks.rateLimit,
12+
RateLimiter: class {},
13+
}))
14+
vi.mock('@/lib/knowledge/application/github-installations', () => ({
15+
listGitHubSearchInstallations: {
16+
operation: { id: 'knowledge.github.installations.list' },
17+
execute: mocks.list,
18+
},
19+
connectGitHubSearchInstallation: {
20+
operation: { id: 'knowledge.github.installations.connect' },
21+
execute: mocks.connect,
22+
},
23+
}))
24+
vi.mock('@/lib/oauth/github-installation', () => ({
25+
GitHubInstallationError: class extends Error {
26+
constructor(
27+
message: string,
28+
readonly status?: number
29+
) {
30+
super(message)
31+
}
32+
},
33+
}))
34+
vi.mock('@/lib/credentials/managed-oauth', () => ({
35+
ManagedOAuthCredentialError: class extends Error {
36+
constructor(
37+
readonly code: string,
38+
message: string,
39+
readonly statusCode: number
40+
) {
41+
super(message)
42+
}
43+
},
44+
}))
45+
46+
import { OrchestrationError } from '@/lib/core/orchestration/types'
47+
import { ManagedOAuthCredentialError } from '@/lib/credentials/managed-oauth'
48+
import { GitHubInstallationError } from '@/lib/oauth/github-installation'
49+
import { GET, POST } from '@/app/api/knowledge/github/installations/route'
50+
51+
const URL = 'http://localhost/api/knowledge/github/installations'
52+
const installation = {
53+
installationId: '123',
54+
accountId: '456',
55+
accountLogin: 'acme',
56+
accountType: 'Organization',
57+
}
58+
59+
beforeEach(() => {
60+
vi.clearAllMocks()
61+
authMockFns.mockGetSession.mockResolvedValue({
62+
user: { id: 'admin-1' },
63+
session: { id: 'session-1' },
64+
})
65+
mocks.rateLimit.mockResolvedValue(null)
66+
mocks.list.mockResolvedValue({
67+
available: true,
68+
installUrl: 'https://github.com/apps/sim-search/installations/new',
69+
needsUserConnection: false,
70+
installations: [installation],
71+
})
72+
mocks.connect.mockResolvedValue({ credential: { id: 'cred-1', displayName: 'GitHub · acme' } })
73+
})
74+
75+
describe('GitHub installation route boundary', () => {
76+
it.each(['GET', 'POST'] as const)(
77+
'authenticates %s before parsing or calling the use case',
78+
async (method) => {
79+
authMockFns.mockGetSession.mockResolvedValue(null)
80+
const request = new NextRequest(URL, method === 'POST' ? { method, body: '{' } : undefined)
81+
const json = vi.spyOn(request, 'json')
82+
const response = await (method === 'GET' ? GET(request) : POST(request))
83+
expect(response.status).toBe(401)
84+
expect(response.headers.get('Cache-Control')).toBe('private, no-store')
85+
expect(json).not.toHaveBeenCalled()
86+
expect(mocks.rateLimit).not.toHaveBeenCalled()
87+
expect(mocks.list).not.toHaveBeenCalled()
88+
expect(mocks.connect).not.toHaveBeenCalled()
89+
}
90+
)
91+
92+
it('applies admission before parsing the POST body', async () => {
93+
mocks.rateLimit.mockResolvedValue(
94+
NextResponse.json({ error: 'Rate limit exceeded' }, { status: 429 })
95+
)
96+
const request = new NextRequest(URL, { method: 'POST', body: '{' })
97+
const json = vi.spyOn(request, 'json')
98+
expect((await POST(request)).status).toBe(429)
99+
expect(json).not.toHaveBeenCalled()
100+
expect(mocks.connect).not.toHaveBeenCalled()
101+
expect(mocks.rateLimit).toHaveBeenCalledWith(
102+
'github-search-installations',
103+
'admin-1',
104+
undefined
105+
)
106+
})
107+
108+
it.each(['0', '-1', '1.5', '123/path', ''])(
109+
'rejects invalid installation ID %s before the use case',
110+
async (installationId) => {
111+
const response = await POST(
112+
new NextRequest(URL, {
113+
method: 'POST',
114+
headers: { 'Content-Type': 'application/json' },
115+
body: JSON.stringify({ organizationId: 'org-1', installationId }),
116+
})
117+
)
118+
expect(response.status).toBe(400)
119+
expect(mocks.connect).not.toHaveBeenCalled()
120+
}
121+
)
122+
123+
it('requires organization scope for GET', async () => {
124+
expect((await GET(new NextRequest(URL))).status).toBe(400)
125+
expect(mocks.list).not.toHaveBeenCalled()
126+
})
127+
128+
it('forwards GET identity and cancellation and projects a private installation list', async () => {
129+
const controller = new AbortController()
130+
const request = new NextRequest(`${URL}?organizationId=org-1`, { signal: controller.signal })
131+
mocks.list.mockResolvedValue({
132+
available: true,
133+
installUrl: 'https://github.com/apps/sim-search/installations/new',
134+
needsUserConnection: false,
135+
installations: [{ ...installation, accessToken: 'private' }],
136+
privateKey: 'private',
137+
})
138+
const response = await GET(request)
139+
expect(response.status).toBe(200)
140+
expect(response.headers.get('Cache-Control')).toBe('private, no-store')
141+
expect(mocks.list).toHaveBeenCalledWith(
142+
expect.objectContaining({
143+
principal: { kind: 'session', userId: 'admin-1', sessionId: 'session-1' },
144+
input: { organizationId: 'org-1', signal: request.signal },
145+
})
146+
)
147+
expect(await response.json()).toEqual({
148+
success: true,
149+
available: true,
150+
installUrl: 'https://github.com/apps/sim-search/installations/new',
151+
needsUserConnection: false,
152+
installations: [installation],
153+
})
154+
})
155+
156+
it('forwards POST cancellation and only returns the safe credential projection', async () => {
157+
const request = new NextRequest(URL, {
158+
method: 'POST',
159+
headers: { 'Content-Type': 'application/json' },
160+
body: JSON.stringify({ organizationId: 'org-1', installationId: '123' }),
161+
})
162+
mocks.connect.mockResolvedValue({
163+
credential: {
164+
id: 'cred-1',
165+
displayName: 'GitHub · acme',
166+
encryptedServiceAccountKey: 'private',
167+
},
168+
created: true,
169+
})
170+
const response = await POST(request)
171+
expect(response.status).toBe(200)
172+
expect(response.headers.get('Cache-Control')).toBe('private, no-store')
173+
expect(mocks.connect).toHaveBeenCalledWith(
174+
expect.objectContaining({
175+
input: { organizationId: 'org-1', installationId: '123', signal: request.signal },
176+
})
177+
)
178+
expect(await response.json()).toEqual({
179+
success: true,
180+
credential: { id: 'cred-1', displayName: 'GitHub · acme' },
181+
})
182+
})
183+
184+
it.each([
185+
[
186+
new OrchestrationError('forbidden', 'Organization administrator access is required'),
187+
403,
188+
'Organization administrator access is required',
189+
],
190+
[
191+
new GitHubInstallationError('Installation permission denied', 403),
192+
403,
193+
'Installation permission denied',
194+
],
195+
[
196+
new GitHubInstallationError('GitHub is temporarily unavailable', 503),
197+
502,
198+
'GitHub is temporarily unavailable',
199+
],
200+
[
201+
new ManagedOAuthCredentialError(
202+
'MANAGED_CREDENTIAL_NEEDS_REAUTH',
203+
'private refresh details',
204+
401
205+
),
206+
401,
207+
'Reconnect your GitHub account to continue installation setup',
208+
],
209+
[new Error('private database details'), 500, 'Internal server error'],
210+
] as const)(
211+
'projects %s without successful installation data',
212+
async (error, status, message) => {
213+
mocks.list.mockRejectedValue(error)
214+
const response = await GET(new NextRequest(`${URL}?organizationId=org-1`))
215+
expect(response.status).toBe(status)
216+
expect(response.headers.get('Cache-Control')).toBe('private, no-store')
217+
const body = await response.json()
218+
expect(body.error).toBe(message)
219+
expect(body).not.toHaveProperty('installations')
220+
expect(body).not.toHaveProperty('credential')
221+
}
222+
)
223+
})
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
import {
2+
connectGitHubSearchInstallationContract,
3+
listGitHubSearchInstallationsContract,
4+
} from '@/lib/api/contracts/knowledge/github-installations'
5+
import {
6+
defineInternalJsonRoute,
7+
extendInternalErrorPolicy,
8+
internalErrorResponse,
9+
internalOrchestrationErrorPolicy,
10+
internalRateLimits,
11+
internalSessionAuth,
12+
} from '@/lib/api/server/routes'
13+
import { ManagedOAuthCredentialError } from '@/lib/credentials/managed-oauth'
14+
import {
15+
connectGitHubSearchInstallation,
16+
listGitHubSearchInstallations,
17+
} from '@/lib/knowledge/application/github-installations'
18+
import { knowledgeOperations } from '@/lib/knowledge/application/operations'
19+
import { GitHubInstallationError } from '@/lib/oauth/github-installation'
20+
21+
const errorPolicy = extendInternalErrorPolicy(internalOrchestrationErrorPolicy, (error) => {
22+
if (error instanceof GitHubInstallationError)
23+
return internalErrorResponse(error.status === 403 ? 403 : 502, { error: error.message })
24+
if (error instanceof ManagedOAuthCredentialError)
25+
return internalErrorResponse(error.statusCode, {
26+
error: 'Reconnect your GitHub account to continue installation setup',
27+
})
28+
return null
29+
})
30+
31+
export const GET = defineInternalJsonRoute({
32+
contract: listGitHubSearchInstallationsContract,
33+
auth: internalSessionAuth,
34+
operation: knowledgeOperations.listGitHubInstallations,
35+
rateLimit: internalRateLimits.user({ bucketName: 'github-search-installations' }),
36+
errorPolicy,
37+
mapInput: ({ query }, { request }) => ({ ...query, signal: request.signal }),
38+
useCase: listGitHubSearchInstallations,
39+
present: (result) => ({ success: true, ...result }),
40+
staticResponseHeaders: { 'Cache-Control': 'private, no-store' },
41+
})
42+
43+
export const POST = defineInternalJsonRoute({
44+
contract: connectGitHubSearchInstallationContract,
45+
auth: internalSessionAuth,
46+
operation: knowledgeOperations.connectGitHubInstallation,
47+
rateLimit: internalRateLimits.user({ bucketName: 'github-search-installations' }),
48+
errorPolicy,
49+
mapInput: ({ body }, { request }) => ({ ...body, signal: request.signal }),
50+
useCase: connectGitHubSearchInstallation,
51+
present: ({ credential }) => ({ success: true, credential }),
52+
staticResponseHeaders: { 'Cache-Control': 'private, no-store' },
53+
})

0 commit comments

Comments
 (0)