603 better oauth datasources #611

NaderRNA · 2024-10-15T22:39:42Z

Ability to connect to datasources (airbyte) using OAuth login page

Uses passport.js to authenticate the OAuth, gets the refreshToken (which is the only thing airbyte needs to establish a connection to a source with OAuth).

Aims to keep current datasource flow as unchanged as possible.

We only have client ID and client Secrets for the following which will be the only ones supported:
(crossed out oauth providers are implemented)

…e api (localhost:8080 to be used instead of the old localhost:8006)

github-actions · 2024-10-15T22:40:45Z

File	Coverage
All files	`96%`	✅
src/lib/utils/validationutils.ts	`95%`	✅

Minimum allowed coverage is 80%

Generated by 🐒 cobertura-action against b4f9ea2

github-advanced-security

ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

…handling of new route to get the redirect url for oauth to be completed

webapp/src/router.ts

-	server.get('/register', unauthedMiddlewareChain, checkSessionWelcome, renderStaticPage(app, '/register'));
+	server.get(
+		'/login',
+		unauthedMiddlewareChain,


webapp/src/router.ts

webapp/src/controllers/airbyte.ts

…ve the internal airbyte api handle it

webapp/src/components/connectorform/DynamicConnectorForm.tsx

…refreshToken

…ill need to patch bugs, fix issues, make dynamic

webapp/src/router.ts

…iltins

webapp/src/components/connectorform/DynamicConnectorForm.tsx

NaderRNA · 2024-10-22T01:59:55Z

Current status of the code:

Successfully authenticates the user with specifically HubSpot currently.
Passes the refreshToken back through the callback function (OAuth controller)
Callback redirects back to datasource/add page with information required to set up datasource
CreateDatasourceForm makes a POST with the sourceConfig set to the required config for the given oauth connector
That then creates the TestDatasource object using the OAuth credentials
Continue datasource creation as standard using the existing flow

The next few commits will likely be refactoring the implementation to do the following in order:

pass the datasource name and datasource description through the flow
Provide the user with better feedback indicating what's happening (adding a new loader for OAuth to say that authentication was successful and Agent Cloud is testing the datasource)
Make the entire flow more compatible for other oauth providers, not just hubspot

webapp/src/controllers/oauth.ts

webapp/src/lib/middleware/auth/fetchsession.ts

webapp/src/lib/oauthsecret/index.ts

webapp/src/controllers/oauth.ts

webapp/src/lib/oauthsecret/index.ts

webapp/src/lib/struct/oauth.ts

webapp/src/controllers/oauth.ts

webapp/src/lib/oauthsecret/index.ts

webapp/src/router.ts

… for dynamic custom passport use

webapp/src/router.ts

+	oauthRouter.get(
+		'/hubspot/callback',
+		useSession,
+		useJWT,


webapp/src/router.ts

+	oauthRouter.get(
+		'/salesforce/callback',
+		useSession,
+		useJWT,


webapp/src/router.ts

+	oauthRouter.get(
+		'/xero/callback',
+		useSession,
+		useJWT,


…tible verify callback

webapp/src/router.ts

+	oauthRouter.get(
+		'/airtable/callback',
+		useSession,
+		useJWT,


NaderRNA · 2024-11-05T03:21:43Z

Somewhere along the line here a bug was introduced that replaced the file upload/drop area with the connectors dropdown

ragyabraham · 2024-11-11T22:36:39Z

@NaderRNA what's happening with this PR?? where are we with this??

NaderRNA · 2024-11-12T00:23:32Z

Currently the branch supports custom OAuth implementation for any OAuth provider where we have a Client ID and Client Secret for the oauth. Currently HubSpot is completley functional using OAuth but there are some issues with out Airtable, Salesforce and Xero app credentials so I'm just trying to iron those out with Andrew.
If you'd like a version that can go to prod alongside BYO VectorDBs I can modify this code so it won't break prod and will work for HubSpot.
But currently if you know the OAuth provider endpoint and required scopes you can create any custom OAuth implementation using this implementation of the code, nothing is hardcoded so introduction of new OAuth implementation is a lot easier for both account creation and datasource auth.

webapp/src/components/CreateDatasourceForm.tsx

webapp/src/controllers/oauth.ts

webapp/src/lib/middleware/passportmanager.ts

This reverts commit cea520e.

webapp/src/controllers/oauth.ts

webapp/src/lib/struct/oauth.ts

NaderRNA · 2024-11-21T04:45:05Z

reverting back because Airtable was creating more problems as I was implementing it. Using custom strategies on passport seems to be broken with our current versions of passport.js. To use airtable or anything not supported by passport.js, a custom implementation is required that skips passport implementation and is completely end to end handled by the webapp

ragyabraham · 2025-05-30T00:59:02Z

@iandjx where are we at with this.... this would be a really good addition

NaderRNA added 2 commits October 10, 2024 11:31

modifying .env.example to fix incorrect port used for internal airbyt…

7036c8c

…e api (localhost:8080 to be used instead of the old localhost:8006)

Merge branch 'develop' into 603-better_oauth_datasources

dd52986

NaderRNA changed the base branch from master to develop October 15, 2024 22:39

adding secret factory for clientId and clientSecret for oauth

0722b51

github-advanced-security bot found potential problems Oct 15, 2024

View reviewed changes

adding new method to airbyte controller, adding api route and router …

3fe4b99

…handling of new route to get the redirect url for oauth to be completed

github-advanced-security bot found potential problems Oct 16, 2024

View reviewed changes

commiting to save progress, fetch non functional returning 405

4f68736

github-advanced-security bot found potential problems Oct 16, 2024

View reviewed changes

webapp/src/controllers/airbyte.ts Fixed Show fixed Hide fixed

webapp/src/controllers/airbyte.ts Fixed Show fixed Hide fixed

NaderRNA added 2 commits October 18, 2024 15:45

refactoring oauth to handle using passport.js instead of trying to ha…

cbde5b4

…ve the internal airbyte api handle it

refactoring redirect, modifying HubSpot OAuth scopes

d56d99b

github-advanced-security bot found potential problems Oct 20, 2024

View reviewed changes

webapp/src/components/connectorform/DynamicConnectorForm.tsx Fixed Show fixed Hide fixed

successful authentication of hubspot, callback url called with valid …

edb07ea

…refreshToken

NaderRNA linked an issue Oct 21, 2024 that may be closed by this pull request

Better OAuth implementation for datasource/airbyte #603

Open

NaderRNA self-assigned this Oct 21, 2024

hubspot successfully integrated into datasource creation workflow, st…

9d3fa15

…ill need to patch bugs, fix issues, make dynamic

github-advanced-security bot found potential problems Oct 22, 2024

View reviewed changes

webapp/src/router.ts Fixed Show fixed Hide fixed

NaderRNA and others added 8 commits October 22, 2024 11:16

migration to add new tool to each team

a9c0ab5

setting the rabbitmq host as the local host ip

d1d9833

adding google and githun auth build args to webapp docker file

a91b518

modifying migration to correctly add tool as option to install for bu…

43b0f0e

…iltins

fixing small naming convention that breaks tool on prod

9dd0842

Use singleton pattern for mongodb and limit pool size to 10

621c41c

Merge branch 'develop' into 603-better_oauth_datasources

e7fdbaa

merging with develop and fixing some front end bugs

7b23881

github-advanced-security bot found potential problems Oct 22, 2024

View reviewed changes

webapp/src/components/connectorform/DynamicConnectorForm.tsx Fixed Show fixed Hide fixed

NaderRNA mentioned this pull request Oct 22, 2024

Custom Passport.js strategies for unsupported OAuth providers #617

Open

NaderRNA linked an issue Oct 22, 2024 that may be closed by this pull request

Custom Passport.js strategies for unsupported OAuth providers #617

Open

initializing salesforce/forcedotcom strategy

96f50af

github-advanced-security bot found potential problems Oct 29, 2024

View reviewed changes

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

adding salesforce and xero passport strategy templates, need more work

3e3ad63

github-advanced-security bot found potential problems Oct 29, 2024

View reviewed changes

fixing mistake with installing npm packages into root instead of webapp

db48429

github-advanced-security bot found potential problems Oct 29, 2024

View reviewed changes

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/lib/oauthsecret/index.ts Fixed Show fixed Hide fixed

webapp/src/lib/struct/oauth.ts Fixed Show fixed Hide fixed

NaderRNA added 3 commits October 30, 2024 09:39

removing node modules

a003104

lint

e0bc5d9

adding slack strategy

b303560

github-advanced-security bot found potential problems Oct 31, 2024

View reviewed changes

NaderRNA added 2 commits November 4, 2024 15:14

adding routes for other strategies

acf79ab

lint

cea520e

github-advanced-security bot found potential problems Nov 4, 2024

View reviewed changes

webapp/src/router.ts Fixed Show fixed Hide fixed

webapp/src/router.ts Fixed Show fixed Hide fixed

NaderRNA added 2 commits November 5, 2024 10:35

Merge branch 'develop' into 603-better_oauth_datasources

f50aaaf

adding custom strategy, starting custom strategy struct, array etc...…

6df80fc

… for dynamic custom passport use

github-advanced-security bot found potential problems Nov 5, 2024

View reviewed changes

successful redirect for airtable oauth, still need to rebuild a compa…

0e14b90

…tible verify callback

github-advanced-security bot found potential problems Nov 5, 2024

View reviewed changes

webapp/src/router.ts

oauthRouter.get(

'/airtable/callback',

useSession,

useJWT,

Check failure

Code scanning / CodeQL

Missing rate limiting High

This route handler performs

authorization

Loading
, but is not rate-limited.

NaderRNA added 3 commits November 21, 2024 10:16

resolving merge conflicts

de9bd1e

fixing issue where file upload wasn't correctly displaying

e26a097

corrctly making airtable request in custom strategy

1d48ae9

github-advanced-security bot found potential problems Nov 21, 2024

View reviewed changes

Revert "lint"

712f9de

This reverts commit cea520e.

github-advanced-security bot found potential problems Nov 21, 2024

View reviewed changes

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/controllers/oauth.ts Fixed Show fixed Hide fixed

webapp/src/lib/struct/oauth.ts Fixed Show fixed Hide fixed

linting the revert

b4f9ea2

603 better oauth datasources #611

Are you sure you want to change the base?

603 better oauth datasources #611

Uh oh!

Conversation

NaderRNA commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Check failure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NaderRNA commented Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check failure

Uh oh!

Check failure

Uh oh!

Check failure

Uh oh!

Check failure

Uh oh!

NaderRNA commented Nov 5, 2024

Uh oh!

ragyabraham commented Nov 11, 2024

Uh oh!

NaderRNA commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NaderRNA commented Oct 15, 2024 •

edited

Loading

github-actions bot commented Oct 15, 2024 •

edited

Loading

NaderRNA commented Oct 22, 2024 •

edited

Loading

NaderRNA commented Nov 12, 2024 •

edited

Loading

NaderRNA commented Nov 21, 2024 •

edited

Loading