Repository navigation
[BR-2165]:feat/add multipart download for shared files #2040
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
fdd28c8
e29819f
2471c96
2f28aab
030a308
6862ab4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -43,22 +43,29 @@ | |
| params: DownloadSharedFileParams | DownloadOwnFileWithMnemonicParams | DownloadOwnFileWithBucketKeyParams, | ||
| ) => DownloadFileResponse; | ||
|
|
||
| const downloadSharedFile: DownloadSharedFileFunction = (params) => { | ||
| const { bucketId, fileId, encryptionKey, token, options } = params; | ||
|
|
||
| return new NetworkFacade( | ||
| /** | ||
| * Creates a NetworkFacade for the bridge. Pass auth for downloads of the user's own files; | ||
| * omit it for shared-link downloads, which authenticate through the share token instead. | ||
| */ | ||
| const createNetworkFacade = (auth?: { username: string; password: string }): NetworkFacade => | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe it's easier to remove the shared download from the network than create an empty auth
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That empty auth was already there before this PR tbh, downloadSharedFile on master did the same thing, I just moved it into createNetworkFacade. And we can't really avoid it, the SDK client requires those fields and for shared files the backend only gives us the key + token, no network creds. The token is what authorizes the download so the auth is ignored anyway. |
||
| new NetworkFacade( | ||
| Network.client( | ||
| envService.getVariable('storjBridge'), | ||
| { | ||
| clientName: 'drive-web', | ||
| clientVersion: packageJson.version, | ||
| }, | ||
| { | ||
| bridgeUser: '', | ||
| userId: '', | ||
| bridgeUser: auth?.username ?? '', | ||
| userId: auth?.password ?? '', | ||
| }, | ||
| ), | ||
| ).download(bucketId, fileId, '', { | ||
| ); | ||
|
|
||
| const downloadSharedFile: DownloadSharedFileFunction = (params) => { | ||
| const { bucketId, fileId, encryptionKey, token, options } = params; | ||
|
|
||
| return createNetworkFacade().download(bucketId, fileId, '', { | ||
| key: Buffer.from(encryptionKey, 'hex'), | ||
| token, | ||
| downloadingCallback: options?.notifyProgress, | ||
|
|
@@ -82,19 +89,7 @@ | |
| } = params; | ||
| const auth = await getAuthFromCredentials(params.creds); | ||
|
|
||
| return new NetworkFacade( | ||
| Network.client( | ||
| envService.getVariable('storjBridge'), | ||
| { | ||
| clientName: 'drive-web', | ||
| clientVersion: packageJson.version, | ||
| }, | ||
| { | ||
| bridgeUser: auth.username, | ||
| userId: auth.password, | ||
| }, | ||
| ), | ||
| ).download(bucketId, fileId, mnemonic, { | ||
| return createNetworkFacade(auth).download(bucketId, fileId, mnemonic, { | ||
| downloadingCallback: options?.notifyProgress, | ||
| abortController: options?.abortController, | ||
| }); | ||
|
|
@@ -109,25 +104,13 @@ | |
| } = params; | ||
| const auth = await getAuthFromCredentials(params.creds); | ||
|
|
||
| return new NetworkFacade( | ||
| Network.client( | ||
| envService.getVariable('storjBridge'), | ||
| { | ||
| clientName: 'drive-web', | ||
| clientVersion: packageJson.version, | ||
| }, | ||
| { | ||
| bridgeUser: auth.username, | ||
| userId: auth.password, | ||
| }, | ||
| ), | ||
| ).downloadWithBucketKey(bucketId, fileId, bucketKey, { | ||
| return createNetworkFacade(auth).downloadWithBucketKey(bucketId, fileId, bucketKey, { | ||
| downloadingCallback: options?.notifyProgress, | ||
| abortController: options?.abortController, | ||
| }); | ||
| }; | ||
|
|
||
| export async function multipartDownload( | ||
| async function multipartDownloadOwnFile( | ||
| params: DownloadOwnFileWithMnemonicParams & { fileSize: number }, | ||
| ): Promise<FileStream> { | ||
| const { | ||
|
|
@@ -139,34 +122,61 @@ | |
| } = params; | ||
| const auth = await getAuthFromCredentials(params.creds); | ||
|
|
||
| const networkFacade = new NetworkFacade( | ||
| Network.client( | ||
| envService.getVariable('storjBridge'), | ||
| { | ||
| clientName: 'drive-web', | ||
| clientVersion: packageJson.version, | ||
| }, | ||
| { | ||
| bridgeUser: auth.username, | ||
| userId: auth.password, | ||
| }, | ||
| ), | ||
| ); | ||
| return new MultipartDownload(createNetworkFacade(auth)).downloadFile({ | ||
| bucketId, | ||
| fileId, | ||
| mnemonic, | ||
| fileSize, | ||
| options: { | ||
| downloadingCallback: options?.notifyProgress, | ||
| abortController: options?.abortController, | ||
| }, | ||
| }); | ||
| } | ||
|
|
||
| const multipartDownload = new MultipartDownload(networkFacade); | ||
| /** | ||
| * Downloads a shared file in chunks. The mnemonic is intentionally empty: shared downloads | ||
| * authenticate through the share token, and options.key overrides the mnemonic-derived | ||
| * decryption key downstream in NetworkFacade.downloadChunk. | ||
| */ | ||
| async function multipartDownloadSharedFile( | ||
|
Check warning on line 142 in src/app/network/download/v2.ts
|
||
| params: DownloadSharedFileParams & { fileSize: number }, | ||
| ): Promise<FileStream> { | ||
| const { bucketId, fileId, encryptionKey, token, fileSize, options } = params; | ||
|
|
||
| return multipartDownload.downloadFile({ | ||
| return new MultipartDownload(createNetworkFacade()).downloadFile({ | ||
| bucketId, | ||
| fileId, | ||
| mnemonic, | ||
| mnemonic: '', | ||
| fileSize, | ||
| options: { | ||
| key: Buffer.from(encryptionKey, 'hex'), | ||
| token, | ||
| downloadingCallback: options?.notifyProgress, | ||
| abortController: options?.abortController, | ||
| }, | ||
| }); | ||
| } | ||
|
|
||
| /** | ||
| * Downloads a file in chunks, dispatching to the shared-link flow when the params carry | ||
| * a share token and encryption key, or to the own-file flow when they carry credentials. | ||
| */ | ||
| export async function multipartDownload( | ||
| params: (DownloadOwnFileWithMnemonicParams | DownloadSharedFileParams) & { fileSize: number }, | ||
| ): Promise<FileStream> { | ||
| const isSharedDownload = Boolean(params.token && params.encryptionKey); | ||
| const isOwnFileDownload = Boolean(params.creds && params.key.mnemonic); | ||
|
|
||
| if (isSharedDownload) { | ||
| return multipartDownloadSharedFile(params as DownloadSharedFileParams & { fileSize: number }); | ||
| } else if (isOwnFileDownload) { | ||
| return multipartDownloadOwnFile(params as DownloadOwnFileWithMnemonicParams & { fileSize: number }); | ||
| } else { | ||
| throw new Error('DOWNLOAD ERRNO. 0'); | ||
| } | ||
| } | ||
|
|
||
| export async function downloadChunkFile( | ||
| params: DownloadOwnFileWithMnemonicParams & { chunkStart: number; chunkEnd: number }, | ||
| ): Promise<FileStream> { | ||
|
|
@@ -180,19 +190,7 @@ | |
| } = params; | ||
| const auth = await getAuthFromCredentials(params.creds); | ||
|
|
||
| return new NetworkFacade( | ||
| Network.client( | ||
| envService.getVariable('storjBridge'), | ||
| { | ||
| clientName: 'drive-web', | ||
| clientVersion: packageJson.version, | ||
| }, | ||
| { | ||
| bridgeUser: auth.username, | ||
| userId: auth.password, | ||
| }, | ||
| ), | ||
| ).downloadChunk({ | ||
| return createNetworkFacade(auth).downloadChunk({ | ||
| bucketId, | ||
| fileId, | ||
| mnemonic, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| import { uploadFile } from './upload'; | ||
| import { downloadFile } from './download'; | ||
| import { downloadFile, multipartDownloadFile } from './download'; | ||
|
|
||
| export default { uploadFile, downloadFile }; | ||
| export default { uploadFile, downloadFile, multipartDownloadFile }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be called with the overrideKey?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont think so, because that's the purpose of the test: to check if the derived key is overridden by the override key. If we changed that line to the override key, it would be checking the same key —
overrideKey === overrideKey.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You added the test because you added the key override to decryptStream. Why was override added to decryptStream?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Answer this @terrerox pls
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sg-gs This PR was clashing with changes I made for new private sharings. So, I've incorporated multi-part downloads into new private sharings: #2116
We can close this PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P.S. It's working, and server changes were already merged, but QA was postponed due to Mail & Mobile
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So shared files can be decrypted in multipart downloads.
Someone opening a shared link doesn't have the owner's mnemonic, so the key the SDK works out is wrong. The share gives us the file's real key as
options.key, and the override tellsdecryptStreamto use it.download()already did this.downloadChunk()didn't, so large shared files would have downloaded corrupted.