Industrial Labs Portal - Comprehensive System Analysis v2

Pattern A: Session-based HTML pages

require_super_admin() checks request.session.get("user") then db.is_super_admin(user_id). Returns redirect to /login if unauthorized.

admin_routes.py:57, dependencies.py:193+

Pattern B: API JSON endpoints

get_current_user(request) + db.is_super_admin(str(user['id'])). Returns HTTPException(403) if unauthorized.

cluster_routes.py:25-28, model_run_routes.py:37-39

Pattern C: Cluster Token machine-to-machine

_validate_cluster_token() checks X-Cluster-Token header against settings.CLUSTER_API_TOKEN.

model_run_routes.py:13-20

Pattern D: API Key Excel Add-in

Authorization: Bearer sk_live_... header. Key SHA-256 hashed for lookup in api_keys.key_hash.

excel_api_routes.py:34-70

RBAC Role Hierarchy

dependencies.py:193+ -- require_role request minimum_role

{
    "id": str(db_user["id"]),           // UUID
    "email": db_user["email"],          // lowercase
    "role": db_user["role"],            // viewer|member|org_admin|super_admin
    "name": db_user["name"],
    "organization_id": str(db_user["organization_id"]),  // UUID
}

Set during Cognito callback at auth_routes.py:401-409. Cleared on logout at auth_routes.py:439.

All RLS policies use the same pattern:

USING (organization_id::text = current_setting('app.current_org_id', true))

Set by middleware on every request. Superuser connection bypasses RLS for admin queries.

All functions in database.py:2541-2663. All use SHA-256 producing 64-char hex.

fund_identity_hash appears in 15 table columns and is the FK hub connecting uploads, versions, posteriors, deliverables, subscriptions, configs, and history.

Full mapping: deep_hash_identity_system.md Sections 4.1-4.7

detect_rename_on_upload() at database.py:2825-2853 checks if a fund_name exists under a different filename in the same org.

resolve_fund_identity_hash() at database.py:2786-2822 follows alias chain with max depth 5 and circular reference protection.

process_and_store_fund_data at database.py:3385-3517

Version numbering scoped to (organization_id, fund_identity_hash). Auto-incremented via MAX(version_number) + 1 with UniqueViolation retry 3 attempts.

save_fund_data_version at database.py:2906-3046

• organization_id: ALWAYS lowercased .lower().strip()
• original_filename: NEVER lowercased only .strip() -- test.xlsx vs Test.xlsx produce DIFFERENT hashes
• fund_name: NEVER lowercased only .strip() -- Fund_1 vs fund_1 produce DIFFERENT hashes
• Migration 005 SQL backfill uses ::bytea cast; Python uses .encode('utf-8') -- could diverge for non-ASCII filenames

database.py:2572-2608, migration 005 SQL

{
  "config": {...},              // Full model configuration
  "fund_data": {                // Dict keyed by fund identifier
    "identity_hash": {
      "contributions": [],
      "distributions": [],
      "quarterly_nav_gp": [],
      "dates": [],
      ...
    }
  },
  "pair_data": {                // Dict keyed by pair identifier
    "pair_id": {
      "market_returns": [],
      "comparable_returns": [],
      "alpha_c": {t: val},
      "beta_c_M": {t: val},
      "h_c_t": {t: {...}},
      ...
    }
  },
  "job_id": "...",
  "metadata": {
    "aws_credentials": {...},
    "org_id": "...",
    "identity_hash_to_fund_info": {...},
    "run_type": "single|evergreen",
    ...
  }
}

run_job.py:37,400-420 structure, run_job.py:591 job_id

run_job.py:1012-1214 routing logic

Portal always forces: override_external=True, weeks_before_batch=9999 always greater than max_time, true_flat_mode + no_priority

run_job.py:1222-1226

cluster_post_processor.py:840-1092 run_parallel_post_processing

S3 Key Patterns:

• Posteriors: posteriors/org_id/fund_identity_hash/job_id.parquet
• Full stats: full_stats/org_id/fund_identity_hash/job_id.parquet
• Client HTML: reports/org_id/fund_identity_hash/job_id_params.html
• Internal HTML: reports/org_id/fund_identity_hash/job_id_internal.html
• Delivery HTML: reports/org_id/fund_identity_hash/job_id_delivery.html

3,096 lines. Prepares external estimates consumed by the NumPyro model as Category 1 priors.

5 GARCH methods in fund_processor: garch, garch_x, garch_corr, egarch, daily_garch_agg

fund_processor.py:1511-1823

PortalLogHandler run_job.py:251-292 bridges Python logger.* calls to the portal API queue.

Category 1 comparable/cashflow: alpha_c, beta_c_M, sigma_c, delta, sigma_d -- use EXTERNAL OLS/GARCH estimates

Category 2 fund-level: alpha, beta, F, sigma_F, lambda, sigma_n -- use INTERNAL posteriors from t-1

model_hierarchical.py:230-369

Init Strategy Fallback Chain: init_to_median then init_to_feasible then init_to_uniform

model_hierarchical.py:5050-5100 fit_funds_jointly, 5350-5400 fit_fund_at_time_t

State Carry 12 elements for single-fund, dict-based for hierarchical:

cum_C, cum_D, cumulative_rate, r_bar_prev, R_t_prev, V_prev,
gp_nav_bar_prev, nav_approx_prev, e_prev, cum_C_minus_D_prev,
r_bar_history, rng_key

Scan invocation: scan transition, init_carry, jnp.arange T

model_hierarchical.py:~2880

Key constraint: scan inside plate = NOT supported. plate inside scan = supported. This dictates the entire model structure.

model_hierarchical.py:18-21

Portal forces true_flat_mode=True + no_priority=True for all non-hierarchical runs.

run_job.py:1222-1226

Common Random Numbers CRN for variance reduction:

• fund_return: max_time x n_samples N 0,1
• distribution: max_time x n_samples N 0,1
• comparable: max_time x n_samples N 0,1
• nav: max_time x n_samples Uniform 0,1 for truncnorm ppf

Memory: ~32 MB for 500 time points x 2000 samples.

model_hierarchical.py:5754-5790 generate_master_shocks

Each fund may start at a different calendar date. align_evergreen_data_to_global_timeline() lines 83-682 aligns all arrays to a shared global timeline by zero-padding the front.

• ZERO_PAD_KEYS: contributions, distributions -- padded with 0.0
• NAN_PAD_KEYS: quarterly_nav_gp, nav_naive, True_V -- padded with NaN
• BOOL_PAD_KEYS: has_distribution, has_nav_gp -- padded with False
• Each fund gets its own aligned pair entry with padded arrays

model_evergreen.py:83-682

compute_evergreen_per_fund_lifecycle() lines 689-813 computes per-fund metadata:

• T_start: First week with non-zero contribution
• T_end: Adjusted for EOL via compute_fund_end_week() from model_hierarchical.py
• T_data_len: Total data array length
• pme_naive_nav: Pre-computed PME-naive NAV for EOL substitution

build_evergreen_at_window() lines 1090-1684 builds the synthetic Fund_EG for a specific window:

1. Active fund set per t -- determine which funds are active
2. Normalizer -- sum of per-fund normalizers
3. Cash flow aggregation -- weighted C, D
4. GP NAV aggregation -- with EOL PME-naive + PP imputation two-pass
5. Weighted comparable returns -- GP-NAV-proportion-weighted
6. OLS on aggregated comparables -- alpha_c, beta_c_M, sigma_c
7. GARCH -- h_c_t conditional variance
8. PME-naive NAV -- from aggregated cash flows
9. True_V / delta / sigma_d / True Fund Returns

model_evergreen.py:1090-1684

The evergreen model passes two critical arrays to create_hierarchical_navcast_model():

• active_mask: shape num_funds x T -- True where fund is active
• fund_entry_times: shape num_funds -- week each fund enters

Inside the hierarchical model transition function:

• Inactive funds: state is frozen no update
• Entry reset: cumulative variables reset when fund first becomes active
• Deterministics recorded AFTER evergreen reset

model_hierarchical.py:4890-4920 freeze/entry-reset logic

deploy-to-asg.sh, deploy-to-ec2.sh at repository root

All portal-to-cluster communication traverses Tailscale VPN:

• SCP: pickle file from portal worker to cluster at /home/ubuntu/jobs/job_id.pkl
• SSH: portal worker SSHes to cluster to execute python run_job.py job_id
• HTTP: cluster POSTs finalize to portal via Tailscale IP
• Presigned S3 URLs: cluster uploads results directly to S3 bypasses VPN for data

cluster_routes.py 3,346 lines provides the admin cluster management interface.

• Fund selection: AG Grid with complex JOINs across upload_funds, fund_identities, fund_data_versions, posterior_results, fund_estimation_history
• Model config: CRUD on model_config_history with config_hash tamper detection
• Run management: Submit, process pending, cancel, retry
• DGP Simulation Lab: Submit simulation jobs with synthetic data
• Cluster.html template: 28,059-line mega-template with AG Grid, charts, estimation panels

portal/app/routers/cluster_routes.py, portal/templates/admin/cluster.html

billing_routes.py:1-687

Also supports custom invoicing via custom_invoices table for billing_type='invoice' orgs.

Key differences from portal upload:

• upload_source='excel_addin'
• Returns JSON not redirect
• Includes fund_hash_map with per-fund hashes
• Versioning failures are HARD errors RuntimeError
• Delivery gate: _check_excel_delivery() checks product_selections for Excel plugin

excel_api_routes.py:34-704

Critical: This is the ONLY path that creates posterior_results and deliverables rows. No retry mechanism -- if finalize fails, S3 files are orphaned.

cluster_post_processor.py:1095-1197 call_portal_finalize 120s timeout