verint.com/google-drive-content-adapter
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Now working as a vm.Script passing in all the Globals the proxy script needs

Browse Source
This commit is contained in:
Peter Morton
2026-03-07 01:20:45 -06:00
parent 67b36f97ce
commit 1a6bd09b7b
4 changed files with 423 additions and 362 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
.specify/memory/constitution.md
View File

@@ -1,35 +1,3 @@
<!--
Sync Impact Report:
Version: 1.11.0 → 1.12.0 (MINOR: Moved Google authentication config to global JSON objects)
Modified Principles:
- Section I.V: Added google-scopes.json to dynamic JSON objects examples
- Removed google.* configuration from config/default.json
- proxy.js Section 1 now reads ONLY from globalThis['service-account-key'] and globalThis['google-scopes']
- config/default.json now contains ONLY infrastructure (server, logging)
- proxy.js authentication is now 100% global-object-based with zero config dependency
Modified Files:
- Created global/google-scopes.json with scopes array setting
- Removed entire google section from config/default.json
- Updated proxy.js Section 1 (initializeServiceAccount) to use only global objects
- Updated server.js validateConfig() to validate new global objects
Breaking Changes:
- config.google.serviceAccountEmail removed (credentials are in global/service-account-key.json)
- config.google.serviceAccountKeyPath removed (always global/service-account-key.json)
- config.google.scopes removed (now in global/google-scopes.json)
- All Google authentication config now exclusively in global/ directory
Modified Sections:
- Section I.V: Updated examples list to include google-scopes
- Section I.IV: Clarified config/default.json contains ONLY infrastructure
Templates Status:
⚠ plan-template.md - Document global/ for ALL authentication and behavioral config
⚠ tasks-template.md - Never put auth, secrets, or behavioral config in config/default.json
✅ spec-template.md - No changes required
Follow-up TODOs:
- Update tests to mock globalThis['service-account-key'] and globalThis['google-scopes']
- Document scopes format in google-scopes.json.example
- Verify all config references removed from proxy.js Section 1
-->
# Proxy Scripts Constitution # Proxy Scripts Constitution
## Core Principles ## Core Principles
@@ -39,41 +7,51 @@ Follow-up TODOs:
**ALL business logic, data processing, authentication, and request handling MUST exist within the `proxy.js` file.** The `server.js` file should ONLY handle: **ALL business logic, data processing, authentication, and request handling MUST exist within the `proxy.js` file.** The `server.js` file should ONLY handle:
- HTTP server setup - HTTP server setup
- Configuration loading - Configuration loading
- Global console replacement with custom logger - Global object injection into isolated context
- Request delegation to `proxy.handleRequest()` - Loading proxy.js via `vm.Script` and `vm.createContext`
- Per-request context creation with all necessary globals
**Rationale**: Monolithic architecture enables simple packaging as a single IVA Studio proxy script and prevents fragmentation of business logic across multiple files. ALL functionality must be in one place. **Implementation via vm.Script**:
`proxy.js` MUST be loaded using Node.js `vm.Script` and executed in isolated contexts created per-request with `vm.createContext`. This ensures:
- Complete isolation from server.js module system
- All dependencies provided explicitly through context objects
- Zero ability to import/export modules
- Pure functional execution with injected dependencies
**Rationale**: Monolithic architecture enables simple packaging as a single IVA Studio proxy script and prevents fragmentation of business logic across multiple files. Using `vm.Script` enforces architectural boundaries at runtime, making it impossible for `proxy.js` to access Node.js module system or file system, ensuring ALL functionality exists in one isolated, dependency-injected file.
### I. Zero External Imports or Exports from `proxy.js` (NON-NEGOTIABLE) ### I. Zero External Imports or Exports from `proxy.js` (NON-NEGOTIABLE)
`proxy.js` MUST have **ZERO import statements**. All dependencies MUST be provided as global objects by server.js. `proxy.js` MUST have **ZERO import statements**. All dependencies MUST be provided through `vm.createContext` by server.js.
`proxy.js` MUST have **ZERO export statements**. The proxy.js file should be treated as a single function that receives (req, res) parameters and loaded as a route into the server.js file. `proxy.js` MUST have **ZERO export statements**. The file MUST be pure JavaScript code executed in an isolated VM context.
**File system access** from `proxy.js` is **ABSOLUTELY PROHIBITED** under any circumstances. The `fs` module MUST NOT be imported into proxy.js. **File system access** from `proxy.js` is **ABSOLUTELY PROHIBITED** under any circumstances. The `fs` module MUST NOT be accessible.
**External libraries** (axios, jwt, googleapis, etc.) MUST NOT be imported. Use globals provided by server.js instead. **External libraries** (axios, jwt, googleapis, etc.) MUST NOT be imported. Dependencies are injected through VM context by server.js.
**Rationale**: Monolithic architecture requires ALL I/O operations and dependency injection to be centralized in server.js, ensuring proxy.js contains ONLY pure business logic. **Rationale**: Using `vm.Script` and `vm.createContext` enforces architectural boundaries at the VM level. proxy.js runs in an isolated context with NO access to Node.js module system, file system, or process globals. ALL dependencies must be explicitly injected per-request through the context object, ensuring proxy.js contains ONLY pure business logic with zero capability for I/O operations.
**For data files that proxy.js needs** (service account keys, certificates, secrets): **For data files that proxy.js needs** (service account keys, certificates, secrets):
1. Place JSON files in `global/` directory 1. Place JSON files in `global/` directory
2. server.js automatically loads them as global objects using the filename as the object name 2. server.js loads them at startup using `loadGlobalObjects()`
3. proxy.js accesses them via `globalThis[objectName]` 3. server.js injects them into VM context per-request via `vm.createContext`
4. proxy.js accesses them as simple variables in context (e.g., `google_drive_settings`)
**Example**: **Example**:
- File: `global/service-account-key.json` - File: `global/google_drive_settings.json`
- Global: `globalThis['service-account-key']` - Loading: server.js reads and assigns to `globalVariableContext.google_drive_settings`
- Access in proxy.js: `const credentials = globalThis['service-account-key']` - Injection: server.js adds `google_drive_settings: globalVariableContext.google_drive_settings` to context
- Access in proxy.js: Direct variable access `google_drive_settings.serviceAccount`
**Enforcement**: **Enforcement**:
- proxy.js MUST have NO `import` statements (file should start with comments, then code) - proxy.js MUST have NO `import` statements (file should start with comments, then code)
- During code review, verify first line of code is NOT an import - proxy.js MUST have NO `export` statements (no module.exports, no export keyword)
- Any `import` statement in proxy.js MUST be rejected immediately - Any `import` or `export` in proxy.js MUST be rejected immediately
- proxy.js MUST have NO `export` statements - server.js MUST load proxy.js using `vm.Script` constructor
- Any `export` statement in proxy.js MUST be rejected immediately - server.js MUST execute via `script.runInContext(context)` with fresh context per request
- All file operations MUST be in server.js, which then provides data via globals - All dependencies injected through `vm.createContext({ ... })` context object
- All external libraries MUST be provided as globals by server.js - VM isolation prevents access to require(), import(), fs, process, and Node.js globals
#### I.I What MUST Be in proxy.js #### I.I What MUST Be in proxy.js
@@ -124,73 +102,109 @@ During code review and planning:
#### I.V Global Objects Provided by server.js #### I.V Global Objects Provided by server.js
The `server.js` file MUST make the following objects available globally for use by `proxy.js`: The `server.js` file MUST inject the following objects into VM context for use by `proxy.js`:
**Core Infrastructure Globals:** **VM Context Injection Pattern:**
```javascript
// Create a context with all globals that proxy.js needs
const context = vm.createContext({
...globalVMContext,
...globalVariableContext,
req,
res,
});
script.runInContext(context);
```
**Core Infrastructure Context Variables:**
1. **console** - Custom logger from `logger.js` 1. **console** - Custom logger from `logger.js`
- Purpose: Structured JSON logging - Purpose: Structured JSON logging
- Usage: `console.info()`, `console.debug()`, `console.error()` - Usage: `console.info()`, `console.debug()`, `console.error()`
- Replaces: Built-in console object - Injected from: `globalVMContext.console`
2. **crypto** - Node.js crypto module 2. **crypto** - Web Crypto API (built-in)
- Purpose: UUID generation, cryptographic operations - Purpose: UUID generation, cryptographic operations
- Usage: `crypto.randomUUID()`, etc. - Usage: `crypto.randomUUID()`, etc.
- Note: Cannot use name 'crypto' due to Web Crypto API conflict - Injected from: `globalVMContext.crypto`
- Replaces: `import crypto from 'node:crypto'` in proxy.js - Note: Web Crypto API available by default in Node.js
3. **config** - Configuration object 3. **config** - Configuration object
- Purpose: Infrastructure settings ONLY (server host/port, logging level) - Purpose: Infrastructure settings ONLY (server host/port, logging level)
- Usage: `global.config.server.port`, `global.config.logging.level` - Usage: `config.server.port`, `config.logging.level`
- Loaded: From `config/default.json` merged with ENV vars - Injected from: `global.config`
- Loaded from: `config/default.json` merged with ENV vars
- **DOES NOT contain**: Authentication, secrets, API keys, behavioral config (use global/ instead) - **DOES NOT contain**: Authentication, secrets, API keys, behavioral config (use global/ instead)
4. **axios** - HTTP client library 4. **axios** - HTTP client library
- Purpose: Making HTTP requests to external APIs - Purpose: Making HTTP requests to external APIs
- Usage: `axios.get(url)`, `axios.post(url, data)` - Usage: `axios.get(url)`, `axios.post(url, data)`
- Package: `axios` - Package: `axios`
- Replaces: `import axios from 'axios'` in proxy.js - Injected from: `globalVMContext.axios`
5. **uuidv4** - UUID v4 generator 5. **uuidv4** - UUID v4 generator
- Purpose: Generate RFC4122 compliant UUIDs - Purpose: Generate RFC4122 compliant UUIDs
- Usage: `uuidv4()` returns string like "110ec58a-a0f2-4ac4-8393-c866d813b8d1" - Usage: `uuidv4()` returns string like "110ec58a-a0f2-4ac4-8393-c866d813b8d1"
- Package: `uuid` (v4 function only) - Package: `uuid` (v4 function only)
- Replaces: `import { v4 as uuidv4 } from 'uuid'` in proxy.js - Injected from: `globalVMContext.uuidv4`
6. **jwt** - JSON Web Token library 6. **jwt** - JSON Web Token library
- Purpose: Creating and verifying JWTs for authentication - Purpose: Creating and verifying JWTs for authentication
- Usage: `jwt.sign(payload, secret)`, `jwt.verify(token, secret)` - Usage: `jwt.sign(payload, secret)`, `jwt.verify(token, secret)`
- Package: `jsonwebtoken` - Package: `jsonwebtoken`
- Replaces: `import jwt from 'jsonwebtoken'` in proxy.js - Injected from: `globalVMContext.jwt`
7. **xmlBuilder** - XML builder/generator 7. **xmlBuilder** - XML builder/generator
- Purpose: Constructing XML documents programmatically - Purpose: Constructing XML documents programmatically
- Usage: `xmlBuilder({ root: { child: 'value' } })` - Usage: `xmlBuilder({ root: { child: 'value' } })`
- Package: `xmlbuilder2` (create function) - Package: `xmlbuilder2` (create function)
- Replaces: `import { create } from 'xmlbuilder2'` in proxy.js - Injected from: `globalVMContext.xmlBuilder`
**Dynamic Data Globals:** **Built-in Web APIs:**
8. **Dynamic JSON objects from global/ directory** 8. **URLSearchParams** - URL query string parser (built-in)
- Purpose: Parse and manipulate URL query strings
- Usage: `new URLSearchParams(queryString)`
- Injected from: `globalVMContext.URLSearchParams`
9. **URL** - URL parser (built-in)
- Purpose: Parse and manipulate URLs
- Usage: `new URL(urlString)`
- Injected from: `globalVMContext.URL`
**Dynamic Data Context Variables:**
10. **Dynamic JSON objects from global/ directory**
- Purpose: Authentication credentials, secrets, API keys, and behavioral configuration - Purpose: Authentication credentials, secrets, API keys, and behavioral configuration
- Pattern: Each `global/filename.json` → `globalThis['filename']` - Pattern: Each `global/filename.json` loaded by server.js → injected into context
- Examples: - Examples:
- `global/service-account-key.json` → `globalThis['service-account-key']` (Service Account credentials with client_email and private_key) - `global/google_drive_settings.json` → context var `google_drive_settings` (consolidated service account, scopes, drive query, sitemap config)
- `global/google-scopes.json` → `globalThis['google-scopes']` (OAuth2 scopes array for Google APIs) - `global/api-keys.json` → context var `api_keys` (API keys and secrets)
- `global/sitemap-config.json` → `globalThis['sitemap-config']` (Sitemap settings like maxUrls) - `global/custom-config.json` → context var `custom_config` (behavioral settings)
- `global/drive-query.json` → `globalThis['drive-query']` (Drive API query filter) - Usage in proxy.js: Direct variable access `google_drive_settings.serviceAccount`
- `global/api-keys.json` → `globalThis['api-keys']` (API keys and secrets) - Loaded: By server.js at startup using `loadGlobalObjects()`
- Usage in proxy.js: `const creds = globalThis['service-account-key']`, `const scopes = globalThis['google-scopes']` - Injected: Per-request via `vm.createContext({ objectName: globalVariableContext.objectName })`
- Loaded: Automatically by server.js at startup using `loadGlobalObjects()`
- **Note**: ALL authentication, secrets, and behavioral configuration MUST be in global/, NEVER in config/default.json - **Note**: ALL authentication, secrets, and behavioral configuration MUST be in global/, NEVER in config/default.json
**Rationale**: Centralizing global setup and ALL file I/O in server.js achieves: **Request/Response Objects:**
- **ZERO imports in proxy.js** - complete dependency injection pattern
- Consistent environment setup and library versions 11. **req** - HTTP IncomingMessage
- Easy testing (mock globals instead of mocking module imports) - Purpose: Access request data (URL, method, headers, body)
- Clear separation: server.js = infrastructure & dependencies, proxy.js = pure business logic - Injected fresh: Per-request from `http.createServer((req, res) => ...)`
- Single source of truth for dependency injection
- Direct REST API calls instead of heavyweight SDK wrappers 12. **res** - HTTP ServerResponse
- Purpose: Send response to client
- Injected fresh: Per-request from `http.createServer((req, res) => ...)`
**Rationale**: Using `vm.createContext` for dependency injection achieves:
- **Runtime-enforced isolation** - proxy.js physically cannot access Node.js module system or file system
- **Zero imports possible** - VM context has no `require()` or `import()` capability
- **Explicit dependencies** - All available objects must be explicitly listed in context
- **Per-request isolation** - Fresh context per request prevents cross-request state leakage
- **Testing simplicity** - Mock entire context object instead of individual module imports
- **Clear contracts** - Context object documents every dependency proxy.js uses
- **Security boundaries** - VM sandbox prevents escape to underlying system
#### I.VI Logging #### I.VI Logging
@@ -380,4 +394,4 @@ All pull requests, code reviews, and design discussions MUST verify compliance w
For runtime development guidance, refer to `.github/prompts/` and `.github/agents/` files which operationalize these principles into agent workflows. For runtime development guidance, refer to `.github/prompts/` and `.github/agents/` files which operationalize these principles into agent workflows.
**Version**: 1.11.0 | **Ratified**: 2026-03-05 | **Last Amended**: 2026-03-07 **Version**: 1.13.0 | **Ratified**: 2026-03-05 | **Last Amended**: 2026-03-07

217
src/proxy.js
View File

@@ -2,10 +2,11 @@
* Google Drive Sitemap Adapter Proxy * Google Drive Sitemap Adapter Proxy
* *
* MONOLITHIC HTTP request handler - ALL functionality in this single file. * MONOLITHIC HTTP request handler - ALL functionality in this single file.
* Architecture: Server.js delegates ALL requests to this module's default function (req, res) => {} * Architecture: Pure IIFE - returns request handler function when executed
* Authentication: Service Account (JWT-based) inline * Authentication: Service Account (JWT-based) inline
* *
* CONSTITUTION REQUIREMENT: ZERO export statements - this file exports ONLY a default handler function * CONSTITUTION REQUIREMENT: ZERO export statements - pure IIFE pattern
* File is loaded by server.js using Function constructor
* *
* Globals provided by server.js: * Globals provided by server.js:
* - console: Custom logger * - console: Custom logger
@@ -59,13 +60,13 @@ function createServiceAccountJWT(credentials, scopes) {
const payload = { const payload = {
iss: credentials.client_email, iss: credentials.client_email,
scope: scopes.join(' '), scope: scopes.join(" "),
aud: 'https://oauth2.googleapis.com/token', aud: "https://oauth2.googleapis.com/token",
exp: expiry, exp: expiry,
iat: now iat: now,
}; };
return jwt.sign(payload, credentials.private_key, { algorithm: 'RS256' }); return jwt.sign(payload, credentials.private_key, { algorithm: "RS256" });
} }
/** /**
@@ -75,12 +76,16 @@ function createServiceAccountJWT(credentials, scopes) {
* @returns {Promise<string>} Access token * @returns {Promise<string>} Access token
*/ */
async function getAccessToken(jwtToken) { async function getAccessToken(jwtToken) {
const response = await axios.post('https://oauth2.googleapis.com/token', { const response = await axios.post(
grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer', "https://oauth2.googleapis.com/token",
assertion: jwtToken {
}, { grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
headers: { 'Content-Type': 'application/x-www-form-urlencoded' } assertion: jwtToken,
}); },
{
headers: { "Content-Type": "application/x-www-form-urlencoded" },
},
);
return response.data.access_token; return response.data.access_token;
} }
@@ -95,19 +100,29 @@ async function getAccessToken(jwtToken) {
async function initializeServiceAccount() { async function initializeServiceAccount() {
try { try {
// Load settings from consolidated global object // Load settings from consolidated global object
const settings = globalThis['google_drive_settings']; const settings = globalThis["google_drive_settings"];
if (!settings) { if (!settings) {
throw new Error('Google Drive settings not found in globalThis["google_drive_settings"]. Ensure server.js loaded global/google_drive_settings.json'); throw new Error(
'Google Drive settings not found in globalThis["google_drive_settings"]. Ensure server.js loaded global/google_drive_settings.json',
);
} }
// Validate service account structure // Validate service account structure
if (!settings.serviceAccount || !settings.serviceAccount.client_email || !settings.serviceAccount.private_key) { if (
throw new Error('Invalid service account key format - missing serviceAccount.client_email or serviceAccount.private_key'); !settings.serviceAccount ||
!settings.serviceAccount.client_email ||
!settings.serviceAccount.private_key
) {
throw new Error(
"Invalid service account key format - missing serviceAccount.client_email or serviceAccount.private_key",
);
} }
// Default scopes if not specified // Default scopes if not specified
const scopes = settings.scopes || ['https://www.googleapis.com/auth/drive.readonly']; const scopes = settings.scopes || [
"https://www.googleapis.com/auth/drive.readonly",
];
// Create JWT token // Create JWT token
const jwtToken = createServiceAccountJWT(settings.serviceAccount, scopes); const jwtToken = createServiceAccountJWT(settings.serviceAccount, scopes);
@@ -115,15 +130,14 @@ async function initializeServiceAccount() {
// Exchange JWT for access token // Exchange JWT for access token
const accessToken = await getAccessToken(jwtToken); const accessToken = await getAccessToken(jwtToken);
console.info('Service account authenticated successfully', { console.info("Service account authenticated successfully", {
email: settings.serviceAccount.client_email email: settings.serviceAccount.client_email,
}); });
return accessToken; return accessToken;
} catch (error) { } catch (error) {
console.error('Service account authentication failed', { console.error("Service account authentication failed", {
error: error.message error: error.message,
}); });
throw error; throw error;
} }
@@ -139,7 +153,7 @@ async function getAccessTokenCached() {
const now = Date.now(); const now = Date.now();
// Return cached token if still valid (with 5 minute buffer) // Return cached token if still valid (with 5 minute buffer)
if (accessTokenCache && tokenExpiryTime && now < (tokenExpiryTime - 300000)) { if (accessTokenCache && tokenExpiryTime && now < tokenExpiryTime - 300000) {
return accessTokenCache; return accessTokenCache;
} }
@@ -180,7 +194,7 @@ function generateRequestId() {
* @returns {boolean} True if valid * @returns {boolean} True if valid
*/ */
function validateDocumentId(id) { function validateDocumentId(id) {
if (!id || typeof id !== 'string') { if (!id || typeof id !== "string") {
return false; return false;
} }
@@ -202,14 +216,14 @@ function validateDocumentId(id) {
* @returns {string} Escaped string safe for XML * @returns {string} Escaped string safe for XML
*/ */
function escapeXml(str) { function escapeXml(str) {
if (!str) return ''; if (!str) return "";
return str return str
.replace(/&/g, '&amp;') .replace(/&/g, "&amp;")
.replace(/</g, '&lt;') .replace(/</g, "&lt;")
.replace(/>/g, '&gt;') .replace(/>/g, "&gt;")
.replace(/"/g, '&quot;') .replace(/"/g, "&quot;")
.replace(/'/g, '&apos;'); .replace(/'/g, "&apos;");
} }
// ============================================================================= // =============================================================================
@@ -238,9 +252,9 @@ class RequestQueue {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
this.queue.push({ handler, resolve, reject }); this.queue.push({ handler, resolve, reject });
console.debug('Request enqueued', { console.debug("Request enqueued", {
queueLength: this.queue.length, queueLength: this.queue.length,
processing: this.processing processing: this.processing,
}); });
// Start processing if not already processing // Start processing if not already processing
@@ -257,15 +271,15 @@ class RequestQueue {
async _processNext() { async _processNext() {
if (this.queue.length === 0) { if (this.queue.length === 0) {
this.processing = false; this.processing = false;
console.debug('Queue empty, stopping processing'); console.debug("Queue empty, stopping processing");
return; return;
} }
this.processing = true; this.processing = true;
const { handler, resolve, reject } = this.queue.shift(); const { handler, resolve, reject } = this.queue.shift();
console.debug('Processing next request', { console.debug("Processing next request", {
remainingInQueue: this.queue.length remainingInQueue: this.queue.length,
}); });
try { try {
@@ -309,7 +323,7 @@ const requestQueue = new RequestQueue();
class DocumentCountExceededError extends Error { class DocumentCountExceededError extends Error {
constructor(count, limit) { constructor(count, limit) {
super(`Document count ${count} exceeds limit of ${limit}`); super(`Document count ${count} exceeds limit of ${limit}`);
this.name = 'DocumentCountExceededError'; this.name = "DocumentCountExceededError";
this.count = count; this.count = count;
this.limit = limit; this.limit = limit;
this.statusCode = 413; this.statusCode = 413;
@@ -333,19 +347,19 @@ class DocumentCountExceededError extends Error {
*/ */
async function queryDocuments(options = {}) { async function queryDocuments(options = {}) {
const { const {
query = 'trashed = false', query = "trashed = false",
fields = 'nextPageToken,files(id,name,mimeType,modifiedTime)', fields = "nextPageToken,files(id,name,mimeType,modifiedTime)",
pageSize = 100, pageSize = 100,
maxDocuments = 50000 maxDocuments = 50000,
} = options; } = options;
const allFiles = []; const allFiles = [];
let pageToken = null; let pageToken = null;
console.debug('Starting Drive API query', { console.debug("Starting Drive API query", {
query, query,
pageSize, pageSize,
maxDocuments maxDocuments,
}); });
const startTime = Date.now(); const startTime = Date.now();
@@ -358,11 +372,11 @@ async function queryDocuments(options = {}) {
const params = new URLSearchParams({ const params = new URLSearchParams({
q: query, q: query,
pageSize: pageSize.toString(), pageSize: pageSize.toString(),
fields fields,
}); });
if (pageToken) { if (pageToken) {
params.append('pageToken', pageToken); params.append("pageToken", pageToken);
} }
// Make direct HTTP call to Drive API // Make direct HTTP call to Drive API
@@ -370,43 +384,41 @@ async function queryDocuments(options = {}) {
`https://www.googleapis.com/drive/v3/files?${params.toString()}`, `https://www.googleapis.com/drive/v3/files?${params.toString()}`,
{ {
headers: { headers: {
'Authorization': `Bearer ${accessToken}`, Authorization: `Bearer ${accessToken}`,
'Accept': 'application/json' Accept: "application/json",
} },
} },
); );
const files = response.data.files || []; const files = response.data.files || [];
allFiles.push(...files); allFiles.push(...files);
console.debug('Drive API page retrieved', { console.debug("Drive API page retrieved", {
pageFiles: files.length, pageFiles: files.length,
totalFiles: allFiles.length, totalFiles: allFiles.length,
hasMore: !!response.data.nextPageToken hasMore: !!response.data.nextPageToken,
}); });
// Check if we've exceeded the limit BEFORE fetching more // Check if we've exceeded the limit BEFORE fetching more
if (allFiles.length > maxDocuments) { if (allFiles.length > maxDocuments) {
console.error('Document count exceeds limit', { console.error("Document count exceeds limit", {
count: allFiles.length, count: allFiles.length,
limit: maxDocuments limit: maxDocuments,
}); });
throw new DocumentCountExceededError(allFiles.length, maxDocuments); throw new DocumentCountExceededError(allFiles.length, maxDocuments);
} }
pageToken = response.data.nextPageToken; pageToken = response.data.nextPageToken;
} while (pageToken); } while (pageToken);
const duration = Date.now() - startTime; const duration = Date.now() - startTime;
console.info('Drive API query completed', { console.info("Drive API query completed", {
documentCount: allFiles.length, documentCount: allFiles.length,
duration duration,
}); });
return allFiles; return allFiles;
} catch (error) { } catch (error) {
// Re-throw DocumentCountExceededError as-is // Re-throw DocumentCountExceededError as-is
if (error instanceof DocumentCountExceededError) { if (error instanceof DocumentCountExceededError) {
@@ -414,10 +426,10 @@ async function queryDocuments(options = {}) {
} }
// Log and re-throw other errors // Log and re-throw other errors
console.error('Drive API query failed', { console.error("Drive API query failed", {
error: error.message, error: error.message,
code: error.code, code: error.code,
statusCode: error.response?.status statusCode: error.response?.status,
}); });
throw error; throw error;
@@ -448,12 +460,12 @@ function mapDriveErrorToHttp(error) {
// Handle rate limiting (429) // Handle rate limiting (429)
if (statusCode === 429) { if (statusCode === 429) {
// Extract Retry-After from response headers if present // Extract Retry-After from response headers if present
const retryAfter = error.response?.headers?.['retry-after']; const retryAfter = error.response?.headers?.["retry-after"];
const retryAfterSeconds = retryAfter ? parseInt(retryAfter, 10) : 60; const retryAfterSeconds = retryAfter ? parseInt(retryAfter, 10) : 60;
return { return {
statusCode: 429, statusCode: 429,
retryAfter: retryAfterSeconds retryAfter: retryAfterSeconds,
}; };
} }
@@ -502,7 +514,7 @@ function validateDocumentCount(count, limit = 50000) {
*/ */
function toSitemapEntry(document, baseUrl) { function toSitemapEntry(document, baseUrl) {
if (!document || !document.id) { if (!document || !document.id) {
console.error('Invalid document for sitemap entry', { document }); console.error("Invalid document for sitemap entry", { document });
return null; return null;
} }
@@ -514,16 +526,16 @@ function toSitemapEntry(document, baseUrl) {
if (document.modifiedTime) { if (document.modifiedTime) {
try { try {
const date = new Date(document.modifiedTime); const date = new Date(document.modifiedTime);
lastmod = date.toISOString().split('T')[0]; // Extract YYYY-MM-DD lastmod = date.toISOString().split("T")[0]; // Extract YYYY-MM-DD
} catch (error) { } catch (error) {
console.error('Invalid modifiedTime for document', { console.error("Invalid modifiedTime for document", {
documentId: document.id, documentId: document.id,
modifiedTime: document.modifiedTime modifiedTime: document.modifiedTime,
}); });
lastmod = new Date().toISOString().split('T')[0]; // Fallback to today lastmod = new Date().toISOString().split("T")[0]; // Fallback to today
} }
} else { } else {
lastmod = new Date().toISOString().split('T')[0]; // Fallback to today lastmod = new Date().toISOString().split("T")[0]; // Fallback to today
} }
return { loc, lastmod }; return { loc, lastmod };
@@ -538,13 +550,13 @@ function toSitemapEntry(document, baseUrl) {
*/ */
function transformDocumentsToSitemapEntries(documents, baseUrl) { function transformDocumentsToSitemapEntries(documents, baseUrl) {
if (!Array.isArray(documents)) { if (!Array.isArray(documents)) {
console.error('Documents must be an array', { documents }); console.error("Documents must be an array", { documents });
return []; return [];
} }
return documents return documents
.map(doc => toSitemapEntry(doc, baseUrl)) .map((doc) => toSitemapEntry(doc, baseUrl))
.filter(entry => entry !== null); .filter((entry) => entry !== null);
} }
/** /**
@@ -561,18 +573,18 @@ function generateSitemapXML(sitemapEntries) {
// Handle empty sitemap - valid XML with no <url> elements // Handle empty sitemap - valid XML with no <url> elements
if (!sitemapEntries || sitemapEntries.length === 0) { if (!sitemapEntries || sitemapEntries.length === 0) {
xml += '</urlset>'; xml += "</urlset>";
return xml; return xml;
} }
for (const entry of sitemapEntries) { for (const entry of sitemapEntries) {
xml += ' <url>\n'; xml += " <url>\n";
xml += ` <loc>${escapeXml(entry.loc)}</loc>\n`; xml += ` <loc>${escapeXml(entry.loc)}</loc>\n`;
xml += ` <lastmod>${escapeXml(entry.lastmod)}</lastmod>\n`; xml += ` <lastmod>${escapeXml(entry.lastmod)}</lastmod>\n`;
xml += ' </url>\n'; xml += " </url>\n";
} }
xml += '</urlset>'; xml += "</urlset>";
return xml; return xml;
} }
@@ -602,20 +614,20 @@ function generateSitemap(documents, baseUrl) {
* @returns {Object} Route info or error * @returns {Object} Route info or error
*/ */
function parseRoute(method, url) { function parseRoute(method, url) {
if (method !== 'GET') { if (method !== "GET") {
return { route: null, error: 'Method not allowed', statusCode: 405 }; return { route: null, error: "Method not allowed", statusCode: 405 };
} }
const urlObj = new URL(url, 'http://localhost'); const urlObj = new URL(url, "http://localhost");
const path = urlObj.pathname; const path = urlObj.pathname;
// Match any path containing 'sitemap.xml' // Match any path containing 'sitemap.xml'
if (path.includes('sitemap.xml')) { if (path.includes("sitemap.xml")) {
return { route: 'sitemap' }; return { route: "sitemap" };
} }
// All other paths return 404 // All other paths return 404
return { route: null, error: 'Not found', statusCode: 404 }; return { route: null, error: "Not found", statusCode: 404 };
} }
/** /**
@@ -629,15 +641,15 @@ function parseRoute(method, url) {
async function handleSitemapRequest(res, requestId) { async function handleSitemapRequest(res, requestId) {
try { try {
// Get configuration from consolidated global settings // Get configuration from consolidated global settings
const settings = globalThis['google_drive_settings'] || {}; const settings = globalThis["google_drive_settings"] || {};
const maxUrls = settings.sitemap?.maxUrls || 50000; const maxUrls = settings.sitemap?.maxUrls || 50000;
const query = settings.driveQuery || 'trashed = false'; const query = settings.driveQuery || "trashed = false";
// Query documents from Drive API // Query documents from Drive API
// This will throw DocumentCountExceededError if exceeds maxUrls limit // This will throw DocumentCountExceededError if exceeds maxUrls limit
const documents = await queryDocuments({ const documents = await queryDocuments({
query: query, query: query,
maxDocuments: maxUrls maxDocuments: maxUrls,
}); });
// Generate sitemap XML with RESTful URLs // Generate sitemap XML with RESTful URLs
@@ -645,16 +657,15 @@ async function handleSitemapRequest(res, requestId) {
// Send successful response // Send successful response
res.statusCode = 200; res.statusCode = 200;
res.setHeader('Content-Type', 'application/xml; charset=utf-8'); res.setHeader("Content-Type", "application/xml; charset=utf-8");
res.setHeader('X-Request-Id', requestId); res.setHeader("X-Request-Id", requestId);
res.setHeader('X-Document-Count', documents.length.toString()); res.setHeader("X-Document-Count", documents.length.toString());
res.end(xml); res.end(xml);
console.info('Sitemap generated successfully', { console.info("Sitemap generated successfully", {
requestId, requestId,
documentCount: documents.length documentCount: documents.length,
}); });
} catch (error) { } catch (error) {
// Map Drive API error to HTTP status code // Map Drive API error to HTTP status code
const errorResponse = mapDriveErrorToHttp(error); const errorResponse = mapDriveErrorToHttp(error);
@@ -663,17 +674,17 @@ async function handleSitemapRequest(res, requestId) {
// Add Retry-After header for rate limiting (429) // Add Retry-After header for rate limiting (429)
if (errorResponse.retryAfter) { if (errorResponse.retryAfter) {
res.setHeader('Retry-After', errorResponse.retryAfter.toString()); res.setHeader("Retry-After", errorResponse.retryAfter.toString());
} }
// Per specification: error responses have NO body // Per specification: error responses have NO body
res.end(); res.end();
console.error('Sitemap generation failed', { console.error("Sitemap generation failed", {
requestId, requestId,
error: error.message, error: error.message,
statusCode: errorResponse.statusCode, statusCode: errorResponse.statusCode,
retryAfter: errorResponse.retryAfter retryAfter: errorResponse.retryAfter,
}); });
} }
} }
@@ -685,14 +696,14 @@ async function handleSitemapRequest(res, requestId) {
* @param {Object} req - HTTP request object * @param {Object} req - HTTP request object
* @param {Object} res - HTTP response object * @param {Object} res - HTTP response object
*/ */
async function handleRequest(req, res) { (async () => {
const requestId = generateRequestId(); const requestId = generateRequestId();
const startTime = Date.now(); const startTime = Date.now();
console.info('Request received', { console.info("Request received", {
requestId, requestId,
method: req.method, method: req.method,
url: req.url url: req.url,
}); });
try { try {
@@ -703,10 +714,10 @@ async function handleRequest(req, res) {
res.statusCode = routeResult.statusCode; res.statusCode = routeResult.statusCode;
res.end(); // Empty body per spec res.end(); // Empty body per spec
console.error('Route not found', { console.error("Route not found", {
requestId, requestId,
url: req.url, url: req.url,
statusCode: routeResult.statusCode statusCode: routeResult.statusCode,
}); });
return; return;
@@ -714,32 +725,28 @@ async function handleRequest(req, res) {
// Handle sitemap route with FIFO queue // Handle sitemap route with FIFO queue
// Per specification: queue concurrent requests, process sequentially // Per specification: queue concurrent requests, process sequentially
if (routeResult.route === 'sitemap') { if (routeResult.route === "sitemap") {
await requestQueue.enqueue(async () => { await requestQueue.enqueue(async () => {
await handleSitemapRequest(res, requestId); await handleSitemapRequest(res, requestId);
}); });
return; return;
} }
} catch (error) { } catch (error) {
res.statusCode = 500; res.statusCode = 500;
res.end(); res.end();
console.error('Request handler error', { console.error("Request handler error", {
requestId, requestId,
error: error.message, error: error.message,
stack: error.stack stack: error.stack,
}); });
} finally { } finally {
const duration = Date.now() - startTime; const duration = Date.now() - startTime;
console.info('Request completed', { console.info("Request completed", {
requestId, requestId,
statusCode: res.statusCode, statusCode: res.statusCode,
duration duration,
}); });
} }
} })();
handleRequest(req, res); // This line is just for clarity - actual invocation is done by server.js

143
src/server.js
View File

@@ -1,54 +1,57 @@
import http from 'node:http'; import http from "node:http";
import { join } from 'node:path'; import { join } from "node:path";
import { readFileSync, readdirSync } from 'node:fs'; import { readFileSync, readdirSync } from "node:fs";
import { fileURLToPath } from 'node:url'; import { fileURLToPath } from "node:url";
import { dirname } from 'node:path'; import { dirname } from "node:path";
import axios from 'axios'; import vm from "node:vm";
import { v4 as uuidv4 } from 'uuid'; import axios from "axios";
import jwt from 'jsonwebtoken'; import { v4 as uuidv4 } from "uuid";
import { create as xmlBuilder } from 'xmlbuilder2'; import jwt from "jsonwebtoken";
import { logger } from './logger.js'; import { create as xmlBuilder } from "xmlbuilder2";
import { logger } from "./logger.js";
const __filename = fileURLToPath(import.meta.url); const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename); const __dirname = dirname(__filename);
// Note: crypto is already available as globalThis.crypto (Web Crypto API) const globalVMContext = {
// No need to import or set it - Node.js provides it by default URLSearchParams,
console: logger,
crypto,
axios,
uuidv4,
jwt,
xmlBuilder,
};
// Replace global console with custom logger let globalVariableContext = {};
globalThis.console = logger;
// Make libraries available globally for proxy.js
globalThis.axios = axios;
globalThis.uuidv4 = uuidv4;
globalThis.jwt = jwt;
globalThis.xmlBuilder = xmlBuilder;
/** /**
* Load all JSON files from global/ directory and make them available as global objects * Load all JSON files from global/ directory and make them available as global objects
* Pattern: global/filename.json -> globalThis['filename'] * Pattern: global/filename.json -> globalVariableContext['filename']
*/ */
function loadGlobalObjects() { function loadGlobalObjects() {
const globalDir = join(__dirname, '..', 'global'); const globalDir = join(__dirname, "..", "global");
try { try {
const files = readdirSync(globalDir).filter(f => f.endsWith('.json') && !f.endsWith('.example')); const files = readdirSync(globalDir).filter(
(f) => f.endsWith(".json") && !f.endsWith(".example"),
);
files.forEach(file => { files.forEach((file) => {
const objectName = file.replace('.json', ''); const objectName = file.replace(".json", "");
const filePath = join(globalDir, file); const filePath = join(globalDir, file);
try { try {
const content = readFileSync(filePath, 'utf-8'); const content = readFileSync(filePath, "utf-8");
const data = JSON.parse(content); const data = JSON.parse(content);
globalThis[objectName] = data; globalVariableContext[objectName] = data;
logger.info(`Loaded global object: ${objectName}`, { logger.info(`Loaded global object: ${objectName}`, {
file: file, file: file,
keys: Object.keys(data) keys: Object.keys(data),
}); });
} catch (error) { } catch (error) {
logger.error(`Failed to load global object from ${file}`, { logger.error(`Failed to load global object from ${file}`, {
error: error.message error: error.message,
}); });
throw error; throw error;
} }
@@ -56,9 +59,9 @@ function loadGlobalObjects() {
logger.info(`Loaded ${files.length} global objects from ${globalDir}`); logger.info(`Loaded ${files.length} global objects from ${globalDir}`);
} catch (error) { } catch (error) {
logger.error('Failed to load global objects', { logger.error("Failed to load global objects", {
directory: globalDir, directory: globalDir,
error: error.message error: error.message,
}); });
throw error; throw error;
} }
@@ -71,12 +74,14 @@ function loadGlobalObjects() {
* @returns {Object} Configuration object * @returns {Object} Configuration object
*/ */
function loadConfig() { function loadConfig() {
const configPath = join(__dirname, '..', 'config', 'default.json'); const configPath = join(__dirname, "..", "config", "default.json");
const configData = readFileSync(configPath, 'utf-8'); const configData = readFileSync(configPath, "utf-8");
const config = JSON.parse(configData); const config = JSON.parse(configData);
// Merge environment variables (ENV vars take precedence) // Merge environment variables (ENV vars take precedence)
config.server.port = process.env.PORT ? parseInt(process.env.PORT, 10) : config.server.port; config.server.port = process.env.PORT
? parseInt(process.env.PORT, 10)
: config.server.port;
config.server.host = process.env.HOST || config.server.host; config.server.host = process.env.HOST || config.server.host;
config.logging.level = process.env.LOG_LEVEL || config.logging.level; config.logging.level = process.env.LOG_LEVEL || config.logging.level;
@@ -92,12 +97,16 @@ function validateConfig(config) {
const errors = []; const errors = [];
// Validate server configuration // Validate server configuration
if (!config.server.port || config.server.port < 1 || config.server.port > 65535) { if (
errors.push('Invalid server.port (must be 1-65535)'); !config.server.port ||
config.server.port < 1 ||
config.server.port > 65535
) {
errors.push("Invalid server.port (must be 1-65535)");
} }
if (errors.length > 0) { if (errors.length > 0) {
throw new Error(`Configuration validation failed:\n${errors.join('\n')}`); throw new Error(`Configuration validation failed:\n${errors.join("\n")}`);
} }
} }
@@ -112,59 +121,73 @@ async function startServer() {
// Load global objects from global/ directory (e.g., service account keys) // Load global objects from global/ directory (e.g., service account keys)
loadGlobalObjects(); loadGlobalObjects();
logger.info('Starting Proxy Script Server...'); logger.info("Starting Proxy Script Server...");
logger.info(`Configuration loaded: ${JSON.stringify({ logger.info(
`Configuration loaded: ${JSON.stringify({
port: global.config.server.port, port: global.config.server.port,
host: global.config.server.host, host: global.config.server.host,
logLevel: global.config.logging.level logLevel: global.config.logging.level,
})}`); })}`,
);
// Validate configuration // Validate configuration
validateConfig(global.config); validateConfig(global.config);
logger.info('Configuration validated successfully'); logger.info("Configuration validated successfully");
// Load proxy.js as a function wrapper (ZERO exports per constitution) const proxyPath = join(__dirname, "proxy.js");
// Import the module which sets up globalThis.handleRequest const proxyCode = readFileSync(proxyPath, "utf-8");
await import('./proxy.js'); const script = new vm.Script(proxyCode, { filename: "proxy.js" });
// Wrap the global handleRequest function for clean invocation
const handleRequest = (req, res) => {
return globalThis.handleRequest(req, res);
};
// Create HTTP server that delegates all requests to proxy // Create HTTP server that delegates all requests to proxy
const server = http.createServer(handleRequest); const server = http.createServer((req, res) => {
try {
// Create a context with all globals that proxy.js needs
const context = vm.createContext({
...globalVMContext,
...globalVariableContext,
req,
res,
});
script.runInContext(context);
} catch (error) {
logger.error("Request handling failed", {
error: error.message,
stack: error.stack,
});
res.statusCode = 500;
res.end("Internal Server Error");
}
});
// Graceful shutdown // Graceful shutdown
const shutdown = () => { const shutdown = () => {
logger.info('\nShutting down gracefully...'); logger.info("\nShutting down gracefully...");
server.close(() => { server.close(() => {
logger.info('Server closed'); logger.info("Server closed");
process.exit(0); process.exit(0);
}); });
// Force shutdown after 10 seconds // Force shutdown after 10 seconds
setTimeout(() => { setTimeout(() => {
logger.error('Forced shutdown after timeout'); logger.error("Forced shutdown after timeout");
process.exit(1); process.exit(1);
}, 10000); }, 10000);
}; };
process.on('SIGTERM', shutdown); process.on("SIGTERM", shutdown);
process.on('SIGINT', shutdown); process.on("SIGINT", shutdown);
// Start listening // Start listening
server.listen(global.config.server.port, global.config.server.host, () => { server.listen(global.config.server.port, global.config.server.host, () => {
logger.info('Server listening', { logger.info("Server listening", {
port: global.config.server.port, port: global.config.server.port,
host: global.config.server.host, host: global.config.server.host,
}); });
}); });
} catch (error) { } catch (error) {
logger.error('Failed to start server', { logger.error("Failed to start server", {
error: error.message, error: error.message,
stack: error.stack stack: error.stack,
}); });
process.exit(1); process.exit(1);
} }

33
tests/unit/utils.test.js
View File

@@ -1,15 +1,20 @@
/** /**
* Unit Tests for General Utilities * Unit Tests for General Utilities
* *
* NOTE: Per constitution requirement, proxy.js has ZERO exports. * NOTE: Per constitution requirement, proxy.js has ZERO exports and NO globalThis usage.
* Internal functions (generateRequestId, validateDocumentId, etc.) cannot be unit tested directly. * The file is a pure function expression loaded via Function constructor.
* These functions are tested indirectly through integration tests of the main handleRequest function.
* *
* This test file verifies constitution compliance only. * This test file verifies constitution compliance only.
*/ */
import { test, describe } from 'node:test'; import { test, describe } from 'node:test';
import assert from 'node:assert'; import assert from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// Set up globals that server.js would provide // Set up globals that server.js would provide
// Note: crypto is already available on globalThis (Web Crypto API) // Note: crypto is already available on globalThis (Web Crypto API)
@@ -17,11 +22,23 @@ globalThis.config = { google: {}, server: {}, sitemap: {} };
describe('Unit: Constitution Compliance', () => { describe('Unit: Constitution Compliance', () => {
test('T046: proxy.js has ZERO exports and exposes handleRequest via globalThis', async () => { test('T046: proxy.js has ZERO exports/imports and loads as pure function', () => {
// Verify proxy.js can be loaded and exposes handleRequest via globalThis const proxyPath = join(__dirname, '..', '..', 'src', 'proxy.js');
await import('../../src/proxy.js'); const proxyCode = readFileSync(proxyPath, 'utf-8');
assert.ok(globalThis.handleRequest, 'handleRequest should be available on globalThis');
assert.strictEqual(typeof globalThis.handleRequest, 'function', 'handleRequest should be a function'); // Verify no exports
assert.ok(!proxyCode.match(/^export /m), 'Should have no export statements');
// Verify no imports
assert.ok(!proxyCode.match(/^import /m), 'Should have no import statements');
// Verify no globalThis usage (except for accessing provided globals)
const globalThisAssignments = proxyCode.match(/globalThis\.[a-zA-Z_]+ =/g);
assert.ok(!globalThisAssignments, 'Should not assign to globalThis');
// Verify it's a function expression that can be executed
assert.ok(proxyCode.includes('(function()'), 'Should contain function expression');
assert.ok(proxyCode.includes('return handleRequest'), 'Should return handleRequest');
}); });
test('T046: crypto is available on globalThis (Web Crypto API)', () => { test('T046: crypto is available on globalThis (Web Crypto API)', () => {