newOS for Developers

File Upload Pipeline

Multi-stage upload system with queue management, S3 integration, and thumbnail generation.

packages/newgraph-signals/src/actions/upload/

Upload Pipeline

The upload process follows a strict sequence:

1. Request Presigned URL→2. PUT to S3→3. Create Post with URL→4. Wait for Processing

Presigned URL Flow

Files are uploaded directly to S3 using presigned URLs. The client never sends file data through the API server—only metadata. This enables large file uploads without API timeout issues.

Upload State Machine

Each upload progresses through these stages:

preparing→created→attached→uploaded

type PostUploadState = {
  status: "preparing" | "created" | "attached" | "uploaded";
  done?: boolean;
  filename: string;
  blob: Blob;           // Original file blob
  thumb: string;        // Base64 thumbnail (images only)
  foldersToAttach: { id: string }[];
};

S3 Presigned URL Flow

Detailed breakdown of the S3 upload mechanism:

Step 1Request Presigned URL

const uploadInfo = await api.post.uploadCreate({
  filename: blob.name,
  targetId: post.id,
  contentType: post.contentType
});
// Returns: { url: "https://s3.../presigned-url" }

Step 2PUT File to S3

await fetch(uploadInfo.data.url, {
  method: "PUT",
  body: blob
});
// File uploaded directly to S3

Step 3Backend Processing

After S3 upload, the backend processes the file (generates thumbnails, extracts metadata, etc.) and updates the post's contentUrl.

Thumbnail Generation

Client-side thumbnail generation for immediate preview.

import { resizeImage } from "../../utils/resizeImage";

// Generate base64 thumbnail for images
const base64thumb = file && 
  file.type.startsWith("image") && 
  (await resizeImage(file));

// Stored in uploadState.thumb for instant preview
const provisionalPost = {
  ...post,
  contentUrl: base64thumb,  // Used as preview
  uploadState: {
    thumb: base64thumb,
    ...
  }
};

Optimistic UI

Thumbnails are generated client-side before upload completes, enabling optimistic UI that shows the image immediately while upload progresses.

createPostSingle

Upload a single file through the full pipeline. Processes items from the upload queue.

createPostSingle()
  : ProgressiveHandlerResponse<PostReadResponse>

// Execute with params:
{ id: string }  // Post ID from upload queue

Parameter	Type	Required	Description
id	string	No	Post ID to process (defaults to last in queue)

Pipeline Steps

Get post ID from queue (or use provided ID)
Execute each stage processor sequentially
Update upload status after each stage
Pop completed item, trigger next job in queue

createPostMultiple

Batch upload multiple files with content mode control.

createPostMultiple()
  : ProgressiveHandlerResponse<string[]>

type CreatePostParams = PostCreateRequest & {
  files: File[];              // Array of files to upload
  foldersToAttach?: string[]; // Target mood IDs
  internalUrls?: string[];    // Optional URL-only posts
  contentMode: "first" | "last" | "each";
};

Parameter	Type	Required	Description
files	File[]	Yes	Array of files to upload
foldersToAttach	string[]	No	Mood IDs to attach posts to
internalUrls	string[]	No	URLs to create as link posts
contentMode	string	Yes	Where to apply text content
content	string	No	Text content for posts

Content Mode

first — Text content only on first file
last — Text content only on last file
each — Text content on every file

Usage Example

import { createPostMultiple } from "newgraph-signals/actions/upload";
import { execProgressiveHandler } from "newgraph-signals";

await execProgressiveHandler(createPostMultiple, {
  files: [file1, file2, file3],
  foldersToAttach: ["mood-123"],
  content: "My gallery upload",
  contentMode: "first"  // Caption on first image only
});

Stage Processors

Located in upload/steps.ts

initializePostUploadEntry Point

Validates file, generates thumbnail, creates provisional post in cache. Sets initial upload state to "preparing".

// Creates provisional post with:
- Generated UUID
- Base64 thumbnail (images)
- Upload state metadata
- Target folder associations

createPostOnServerpreparing → created

Creates post in API with contentUrl: "preparing". Handles "future" folders by creating them first.

updateAndAttachcreated → attached

Attaches post to target Mood(s) via attachToFolders().

uploadFileattached → uploaded

Requests presigned S3 URL, uploads file via PUT.

Upload Progress Tracking

Track upload progress using reactive signals and cache state.

import { uploadQueueSignal } from "newgraph-signals/actions/upload";
import { cache, EnrichedPost } from "newgraph-signals";

// Monitor queue length
effect(() => {
  console.log("Pending uploads:", uploadQueueSignal.value.length);
});

// Get individual post upload status
const post = await cache.post.get(postId) as EnrichedPost;
console.log(post.uploadState?.status);
// "preparing" | "created" | "attached" | "uploaded"

Reactive Signals

uploadQueueSignalSignal<string[]>

Queue of post IDs pending upload. Processed right-to-left (LIFO). New items added to front, processed from end.

uploadResponseSignalSignal<PostReadResponse>

Latest uploaded post response.

Helper Functions

Located in upload/helpers.ts

updateUploadStatus(status, post, folders)

Update post upload state in cache. Updates uploadState.statusand refreshes updated timestamp.

uuidv4()

Generate unique ID for new uploads. Uses crypto.getRandomValues for randomness. Retries up to 3 times if generation fails.

retry<T>(fn: () => Promise<T>)

Retry failed operations with exponential backoff (300ms, 600ms, 900ms, 1500ms).

wait(ms: number)

Promise-based delay utility for rate limiting between operations.

Desktop Integration

Electron Support

The upload module listens for window.postMessage events for Electron desktop app integration. Posts sent via message are automatically queued for upload.

// Electron sends upload via postMessage
window.postMessage({
  foldersToAttach: ["mood-123"],
  content: "From desktop",
  files: [...]
}, "*");

// Also listens for WebSocket notifications
websocketEvents.on("newgraph-notification", (ev) => {
  if (ev?.type === "newgraph" && 
      ev.payload.message === "post_in_folder") {
    // Handle real-time post updates
  }
});

Error Handling

No Content Error

"post-create: Post must have either content or an attached file"

Unsupported Type Error

"post-create: Unrecognized/unsupported content type. Upload something else."

S3 Upload Error

"post-create: The post was created but couldn't upload the file"

Underlying API

api.post.postCreate(postData)

POST — Create post record with contentUrl: "preparing"

api.post.uploadCreate({ filename, targetId, contentType })

GET — Returns presigned S3 URL for direct upload