From 3e4c673593c56585b31e085e4f84201bd12750b4 Mon Sep 17 00:00:00 2001 From: zhendi Date: Fri, 30 May 2025 09:09:32 +0800 Subject: [PATCH 1/6] feat(docs): add comprehensive documentation - Add User and Authentication System - Add User Authentication and Session Management - Add User Profile and Settings - Add Organizations and Collections - Add Frontend Architecture - Add Component System and Navigation - Add Internationalization (i18n) - Add API Integration and Data Fetching - Add Admin Panel - Add Backend Architecture - Add CSGHub Overview - Add Routing and Middleware - Add Repository Handlers - Add Development and Deployment - Add Repository Management System - Add Repository Detail Components - Add Repository Clone and Actions - Add Repository Browsing and Filtering - Add Model Deployment and Fine-tuning - Add Creating and Managing Endpoints - Add Model Fine-tuning - Add Model Settings and Evaluation --- .../10_User_and_Authentication_System.md | 322 +++++++++++ ...r_Authentication_and_Session_Management.md | 278 ++++++++++ .../12_User_Profile_and_Settings.md | 369 ++++++++++++ .../13_Organizations_and_Collections.md | 272 +++++++++ docs/development/14_Frontend_Architecture.md | 413 ++++++++++++++ .../15_Component_System_and_Navigation.md | 491 ++++++++++++++++ .../16_Internationalization_i18n.md | 292 ++++++++++ .../17_API_Integration_and_Data_Fetching.md | 298 ++++++++++ docs/development/18_Admin_Panel.md | 313 +++++++++++ docs/development/19_Backend_Architecture.md | 451 +++++++++++++++ docs/development/1_CSGHub_Overview.md | 457 +++++++++++++++ docs/development/20_Routing_and_Middleware.md | 317 +++++++++++ docs/development/21_Repository_Handlers.md | 307 ++++++++++ .../22_Development_and_Deployment.md | 317 +++++++++++ .../2_Repository_Management_System.md | 494 ++++++++++++++++ .../3_Repository_Detail_Components.md | 366 ++++++++++++ .../4_Repository_Clone_and_Actions.md | 280 ++++++++++ .../5_Repository_Browsing_and_Filtering.md | 346 ++++++++++++ .../6_Model_Deployment_and_Fine-tuning.md | 525 ++++++++++++++++++ .../7_Creating_and_Managing_Endpoints.md | 350 ++++++++++++ docs/development/8_Model_Fine-tuning.md | 269 +++++++++ .../9_Model_Settings_and_Evaluation.md | 407 ++++++++++++++ 22 files changed, 7934 insertions(+) create mode 100644 docs/development/10_User_and_Authentication_System.md create mode 100644 docs/development/11_User_Authentication_and_Session_Management.md create mode 100644 docs/development/12_User_Profile_and_Settings.md create mode 100644 docs/development/13_Organizations_and_Collections.md create mode 100644 docs/development/14_Frontend_Architecture.md create mode 100644 docs/development/15_Component_System_and_Navigation.md create mode 100644 docs/development/16_Internationalization_i18n.md create mode 100644 docs/development/17_API_Integration_and_Data_Fetching.md create mode 100644 docs/development/18_Admin_Panel.md create mode 100644 docs/development/19_Backend_Architecture.md create mode 100644 docs/development/1_CSGHub_Overview.md create mode 100644 docs/development/20_Routing_and_Middleware.md create mode 100644 docs/development/21_Repository_Handlers.md create mode 100644 docs/development/22_Development_and_Deployment.md create mode 100644 docs/development/2_Repository_Management_System.md create mode 100644 docs/development/3_Repository_Detail_Components.md create mode 100644 docs/development/4_Repository_Clone_and_Actions.md create mode 100644 docs/development/5_Repository_Browsing_and_Filtering.md create mode 100644 docs/development/6_Model_Deployment_and_Fine-tuning.md create mode 100644 docs/development/7_Creating_and_Managing_Endpoints.md create mode 100644 docs/development/8_Model_Fine-tuning.md create mode 100644 docs/development/9_Model_Settings_and_Evaluation.md diff --git a/docs/development/10_User_and_Authentication_System.md b/docs/development/10_User_and_Authentication_System.md new file mode 100644 index 000000000..5d8516b8a --- /dev/null +++ b/docs/development/10_User_and_Authentication_System.md @@ -0,0 +1,322 @@ +# User and Authentication System + +This document provides a comprehensive overview of the user management and authentication architecture in CSGHub. It covers the authentication flow, session management, user roles, and profile management functionality. + +For information on organization management and collections, see [Organizations and Collections](#4.3). + +## 1. System Overview + +The User and Authentication System in CSGHub is responsible for managing user identities, authentication, sessions, and access control. It implements a JWT (JSON Web Token)-based authentication mechanism with role-based access control. + +```mermaid +flowchart TD + subgraph "Frontend" + Navbar["Navbar.vue"] + UserStore["UserStore (Pinia)"] + ProfileComponents["Profile Components"] + AuthUtils["auth.js utilities"] + end + + subgraph "Backend" + SessionHandler["SessionHandler"] + UserModel["User Model"] + AuthMiddleware["Auth Middleware"] + JWT["JWT Verification"] + end + + subgraph "Persistence" + Cookies["Browser Cookies"] + LocalStorage["Local Storage"] + Database[(Database)] + end + + User((User)) --> Navbar + Navbar --> UserStore + ProfileComponents --> UserStore + + UserStore --> AuthUtils + AuthUtils --> SessionHandler + + SessionHandler --> JWT + JWT --> UserModel + UserModel --> Database + + SessionHandler --> Cookies + UserStore --> LocalStorage + + AuthMiddleware --> JWT + AuthMiddleware --> UserModel +``` + +Sources: [frontend/src/components/navbar/Navbar.vue](), [frontend/src/stores/UserStore.js](), [internal/handlers/render/session.go](), [internal/models/user.go]() + +## 2. User Model and Database Schema + +The user model defines the structure and behavior of user entities in the system, including their roles and permissions. + +### 2.1. Core User Entity + +The `User` struct is the primary data structure for user information: + +```mermaid +classDiagram + class User { + +int64 ID + +string Nickname + +string Name + +string Email + +string LoginIdentity + +string Gender + +int RolesMask + +string Phone + +string SessionIP + +string Avatar + +time CreatedAt + +time UpdatedAt + +Roles() []string + +HasRole(role string) bool + +IsAdmin() bool + +IsSuperUser() bool + +SetRoles(roles ...string) + } +``` + +Sources: [internal/models/user.go:19-31](), [pkg/database/migrations/20240902082008_create_admin_photo.go:9-22]() + +### 2.2. Role-Based Access Control + +The system implements role-based access control using a bitmask approach: + +| Role Constant | Bitmask Value | Description | +|---------------|---------------|-------------| +| `RoleSuperUser` | 1 (1 << 0) | Super user with full access | +| `RoleAdmin` | 2 (1 << 1) | Admin with administrative access | +| `RolePersonalUser` | 4 (1 << 2) | Standard personal user | +| `RoleCompanyUser` | 8 (1 << 3) | Corporate user | + +The `User` struct provides methods to check and manage roles: +- `Roles()`: Returns all roles as string array +- `HasRole(role)`: Checks if user has specific role +- `IsAdmin()`: Checks if user has admin privileges +- `IsSuperUser()`: Checks if user has super user privileges + +Sources: [internal/models/user.go:33-94]() + +## 3. Authentication Flow + +The authentication process in CSGHub utilizes JWT (JSON Web Token) for secure user authentication. + +```mermaid +sequenceDiagram + participant User as "User" + participant Frontend as "Frontend" + participant SessionHandler as "SessionHandler" + participant JWT as "JWT Verifier" + participant DB as "User Database" + + User->>Frontend: Click Login + Frontend->>SessionHandler: Redirect to Login URL + SessionHandler->>User: Present Login Page + User->>SessionHandler: Enter Credentials + SessionHandler->>JWT: Verify JWT Token + JWT->>SessionHandler: Return User Info + + SessionHandler->>DB: Find User by LoginIdentity + alt User Exists + DB->>SessionHandler: Return User Data + SessionHandler->>DB: Update User Info + else User Does Not Exist + SessionHandler->>DB: Create New User + end + + SessionHandler->>Frontend: Set Cookies (user_token, login_identity) + SessionHandler->>Frontend: Redirect to Home/Previous Page + Frontend->>UserStore: Initialize User Data +``` + +Sources: [internal/handlers/render/session.go:71-151]() + +### 3.1. Login Process + +1. User initiates login by clicking the login button in the navbar +2. User is redirected to the login URL defined in the configuration +3. After successful authentication, a JWT token is generated +4. The `SessionHandler.Create` method verifies the JWT token +5. The system creates a new user record if the user doesn't exist, or updates existing user information +6. User session information is stored in cookies +7. User is redirected to the home page or a previously accessed page + +Sources: [internal/handlers/render/session.go:47-69](), [internal/handlers/render/session.go:71-151]() + +### 3.2. Logout Process + +1. User initiates logout from the navbar dropdown +2. The `clearCookies` method in the Navbar component calls the logout function +3. All cookies are removed, including login tokens +4. User is redirected to the home page + +Sources: [frontend/src/components/navbar/Navbar.vue:214-222](), [frontend/src/components/navbar/Navbar.vue:379-382](), [internal/handlers/render/session.go:51-65]() + +## 4. Session Management + +CSGHub manages user sessions through cookies and local storage persistence. + +### 4.1. Cookie Management + +The system uses several cookies to manage user sessions: + +| Cookie Name | Purpose | Expiration | +|-------------|---------|------------| +| `user_token` | Stores the JWT token | 7 days | +| `login_identity` | Stores the user's unique identifier (UUID) | 7 days | +| `can_change_username` | Indicates if the user can change their username | 7 days | +| `previous_path` | Stores the last visited path for redirection after login | Session | +| `locale` | Stores the user's language preference | Persistent | + +Sources: [internal/handlers/render/session.go:141-143](), [frontend/src/components/navbar/Navbar.vue:351-352]() + +### 4.2. Frontend State Persistence + +The frontend uses Pinia with a custom persistence plugin to maintain user state across page reloads: + +1. User data is stored in the browser's local storage with a 2-minute expiration time +2. On page load, the system checks if the stored state is still valid +3. If valid, the user state is restored; otherwise, it's cleared +4. This persistence layer works alongside cookies to provide a seamless user experience + +Sources: [frontend/src/packs/persistPinia.js:1-20](), [frontend/src/stores/UserStore.js:89-93]() + +## 5. User Store and Frontend Integration + +The frontend maintains user state using Pinia, a state management library for Vue.js. + +```mermaid +flowchart TD + subgraph "UserStore Components" + Store["UserStore (Pinia)"] + State["User State\n- username\n- email\n- avatar\n- roles\n- etc."] + ComputedProps["Computed Properties\n- isLoggedIn\n- isAdmin\n- hasEmail\n- etc."] + Actions["Actions\n- initialize()\n- clearStore()\n- etc."] + end + + subgraph "UI Components" + Navbar["Navbar.vue"] + Profile["Profile.vue"] + ProfileEdit["ProfileEdit.vue"] + Settings["ProfileSettings.vue"] + end + + Store --> State + Store --> ComputedProps + Store --> Actions + + Navbar --> |"Uses"| Store + Profile --> |"Uses"| Store + ProfileEdit --> |"Uses"| Store + Settings --> |"Uses"| Store + + ComputedProps --> |"Controls UI visibility"| Navbar +``` + +Sources: [frontend/src/stores/UserStore.js:7-93](), [frontend/src/components/navbar/Navbar.vue:322-323](), [frontend/src/components/navbar/Navbar.vue:361]() + +### 5.1. Key Features of the User Store + +1. **Basic User Data Storage**: + - Personal information: username, nickname, email, phone + - Profile data: avatar, homepage, bio + - System information: uuid, roles, timestamps + +2. **Computed Properties for Access Control**: + - `isLoggedIn`: Determines if user is authenticated + - `isAdmin`: Checks if user has admin privileges + - `isSuperUser`: Checks if user has super user privileges + - `actionLimited`: Determines if user actions should be restricted (e.g., missing email or required username change) + +3. **Persistence**: + - User data is persisted in local storage + - Short expiration time (2 minutes) to ensure data freshness + - Synchronized with server on page load + +Sources: [frontend/src/stores/UserStore.js:7-93](), [frontend/src/packs/persistPinia.js:1-20]() + +## 6. User Interface Components + +The user interface for authentication and profile management is primarily implemented through several Vue components. + +### 6.1. Navbar Component + +The Navbar component is the main entry point for user authentication interactions: + +1. Displays login/register button for unauthenticated users +2. Shows user avatar and dropdown menu for authenticated users +3. Provides access to user profile, settings, and other user-specific features +4. Handles logout process +5. Displays notifications for missing email or required username changes + +Sources: [frontend/src/components/navbar/Navbar.vue:1-443]() + +### 6.2. Profile Management Components + +Several components handle user profile management: + +| Component | Purpose | +|-----------|---------| +| `Profile.vue` | Displays user profile information | +| `ProfileEdit.vue` | Allows editing of user profile data | +| `ProfileSettings.vue` | Manages user settings | +| `AccessTokenSettings.vue` | Manages API access tokens | +| `SshKeySettings.vue` | Manages SSH keys for Git operations | + +These components are registered in the main application entry point and rendered based on the current route. + +Sources: [frontend/src/main.js:26-28](), [frontend/src/main.js:30-31]() + +## 7. System Configuration and Initialization + +The authentication system is initialized during application startup. + +### 7.1. Backend Initialization + +1. The application loads configuration via `config.LoadConfig()` +2. The database connection is initialized through `models.InitDB()` +3. A service context is created with `svc.NewServiceContext()` +4. Routes are initialized including authentication middleware +5. The HTTP server is started to handle requests + +Sources: [cmd/csghub-portal/cmd/start/server.go:15-51](), [internal/models/common.go:15-22]() + +### 7.2. Frontend Initialization + +1. The main application is created using Vue.js +2. Pinia store is initialized with custom persistence plugin +3. User components are registered +4. Internationalization (i18n) is set up with cookie-based locale detection +5. Router is configured for navigation +6. The application is mounted to the DOM + +Sources: [frontend/src/main.js:1-168](), [frontend/src/admin.js:1-71]() + +## 8. Admin User Management + +The system includes specialized admin interfaces for user management: + +1. Admin panel accessible only to users with admin privileges +2. User listing and management capabilities +3. System configuration settings +4. Additional management features for admin users + +Sources: [frontend/src/main.js:61-69](), [frontend/src/components/navbar/Navbar.vue:137-145]() + +## Summary + +The User and Authentication System in CSGHub provides a comprehensive solution for user management, authentication, and authorization. It uses JWT for secure authentication, implements role-based access control, and provides a seamless user experience through persistent state management. + +Key features include: +- JWT-based authentication +- Role-based access control +- Cookie and local storage persistence +- User profile management +- Admin interfaces for user administration + +This system forms the foundation for user interactions throughout the CSGHub platform, ensuring secure and personalized access to resources. \ No newline at end of file diff --git a/docs/development/11_User_Authentication_and_Session_Management.md b/docs/development/11_User_Authentication_and_Session_Management.md new file mode 100644 index 000000000..7f92f8522 --- /dev/null +++ b/docs/development/11_User_Authentication_and_Session_Management.md @@ -0,0 +1,278 @@ +# User Authentication and Session Management + +This document provides a technical overview of the authentication system and session management implementation in CSGHub. It covers the authentication flow, JWT token handling, user session management, and the role-based access control mechanisms. For information about user profiles and settings, see [User Profile and Settings](#4.2). + +## 1. Authentication Architecture Overview + +CSGHub implements a cookie-based authentication system using JWT (JSON Web Tokens) for secure user identity verification. The system maintains user state between sessions using browser cookies and client-side state management. + +```mermaid +flowchart TD + subgraph "Frontend" + A["User Interface"] --> B["Login Button"] + B --> C["External Auth Provider"] + C --> D["JWT Token Callback"] + D --> E["Cookie Storage"] + E --> F["UserStore (Pinia)"] + F --> G["NavBar Component"] + F --> H["Protected Routes"] + end + + subgraph "Backend" + I["SessionHandler"] --> J["JWT Verification"] + J --> K["User Database"] + K --> L["Role Assignment"] + L --> M["Session Response"] + end + + C --> I + M --> D +``` + +Sources: +- [frontend/src/components/navbar/Navbar.vue:60-225]() +- [internal/handlers/render/session.go:18-151]() +- [frontend/src/stores/UserStore.js:1-95]() + +## 2. Authentication Flow + +### 2.1 Login Process + +1. The user initiates login from the Navbar component +2. The system redirects to the external login page (defined in config) +3. After successful authentication, the external system redirects back with a JWT token +4. The SessionHandler verifies the JWT and creates/updates the user record +5. Cookies are set for maintaining the session +6. The user is redirected to the requested page or home page + +```mermaid +sequenceDiagram + participant User + participant UI as "Frontend UI" + participant AuthProvider as "Auth Provider" + participant SessionHandler as "SessionHandler" + participant DB as "User Database" + + User->>UI: Click Login + UI->>AuthProvider: Redirect to login page + AuthProvider->>User: Present login form + User->>AuthProvider: Enter credentials + AuthProvider->>SessionHandler: Redirect with JWT token + SessionHandler->>SessionHandler: Verify JWT token + SessionHandler->>DB: Find user by login_identity + alt User exists + SessionHandler->>DB: Update user info + else User doesn't exist + SessionHandler->>DB: Create new user + end + SessionHandler->>UI: Set cookies and redirect + UI->>UI: Initialize UserStore +``` + +Sources: +- [internal/handlers/render/session.go:47-49]() - Login handler +- [internal/handlers/render/session.go:71-151]() - Session creation and JWT verification + +### 2.2 JWT Token Handling + +The JWT token contains user identity information and roles. The backend verifies this token for authenticity and then uses it to: + +1. Identify the user +2. Determine user roles and permissions +3. Maintain session state + +The token is stored in cookies: +- `user_token`: Contains the actual JWT token +- `login_identity`: Contains the user's UUID +- `can_change_username`: Indicates if the user needs to update their username + +Sources: +- [internal/handlers/render/session.go:80-93]() - JWT token verification +- [internal/handlers/render/session.go:141-143]() - Setting cookies with user information + +## 3. Session Management + +### 3.1 Cookie-Based Session + +CSGHub uses HTTP cookies to maintain user sessions with the following characteristics: + +- **Cookie Duration**: 7 days (defined by `cookieMaxAge = 3600 * 24 * 7`) +- **Cookie Types**: + - `user_token`: The JWT token + - `login_identity`: User UUID + - `can_change_username`: Username modification flag + - `locale`: User's preferred language + +```mermaid +classDiagram + class SessionHandlerImpl { + +Login(ctx *gin.Context) + +Logout(ctx *gin.Context) + +SignUp(ctx *gin.Context) + +Create(ctx *gin.Context) + } + + class UserStore { + +username: string + +nickname: string + +email: string + +roles: array + +uuid: string + +initialize(data) + +clearStore() + +refreshCanChangeUsernameCookie() + } + + SessionHandlerImpl --> UserStore : "manages user data for" +``` + +Sources: +- [internal/handlers/render/session.go:18-29]() - Cookie constants and SessionHandler interface +- [frontend/src/stores/UserStore.js:1-95]() - Frontend session state management + +### 3.2 Logout Process + +The logout process: +1. Clears all cookies by setting them with a negative expiration time +2. Clears the UserStore state using the clearStore action +3. Redirects the user to the home page or a specified redirect path + +Sources: +- [internal/handlers/render/session.go:51-65]() - Backend logout handler +- [frontend/src/components/navbar/Navbar.vue:379-382]() - Frontend logout function + +### 3.3 Session State Management + +The frontend uses Pinia for state management with the following features: + +- **Persistence**: Session state is saved to localStorage with a 2-minute expiration +- **State Properties**: User details, roles, login status, and action limitations +- **Computed Properties**: Authorization flags like `isAdmin`, `isLoggedIn`, `actionLimited` + +Sources: +- [frontend/src/stores/UserStore.js:7-93]() - UserStore definition +- [frontend/src/packs/persistPinia.js:1-20]() - State persistence implementation + +## 4. User Model and Roles + +### 4.1 User Database Model + +The User model stores essential user information: + +```mermaid +erDiagram + User { + int64 ID PK + string Nickname + string Name + string Email + string LoginIdentity PK + string Gender + int RolesMask + string Phone + string SessionIP + datetime CreatedAt + datetime UpdatedAt + } +``` + +Sources: +- [internal/models/user.go:19-31]() - User struct definition +- [pkg/database/migrations/20240902082008_create_admin_photo.go:9-22]() - User database migration + +### 4.2 Role-Based Access Control + +CSGHub implements bit-masked roles for flexible permission management: + +| Role | Bit Value | Description | +|------|-----------|-------------| +| Super User | 1 (1 << 0) | Highest level system administrator | +| Admin | 2 (1 << 1) | Platform administrator | +| Personal User | 4 (1 << 2) | Standard individual user | +| Company User | 8 (1 << 3) | Business/organizational user | + +Role checks are implemented in both backend and frontend: + +- Backend: Using bit operations on the `RolesMask` field +- Frontend: Using computed properties from UserStore + +The frontend UI adapts based on the user's roles, showing or hiding specific menu items and functionality. + +Sources: +- [internal/models/user.go:33-94]() - Role definitions and helper methods +- [frontend/src/stores/UserStore.js:26-33]() - Frontend role computation +- [frontend/src/components/navbar/Navbar.vue:137-145]() - Admin panel access control + +## 5. Frontend Implementation Details + +### 5.1 Navbar and User Menu + +The Navbar component displays different UI elements based on the user's authentication state: + +- **Logged Out**: Shows login/register button +- **Logged In**: Shows user avatar and dropdown menu with: + - Profile link + - Settings + - Resource console (if not action limited) + - Admin panel (for admins) + - Repository creation options + - Logout option + +The component also displays warning banners when: +- Email is missing +- Username needs to be changed +- Both email and username need attention + +Sources: +- [frontend/src/components/navbar/Navbar.vue:60-225]() - Avatar dropdown implementation +- [frontend/src/components/navbar/Navbar.vue:287-315]() - Warning banners + +### 5.2 User Store Integration + +The UserStore is integrated throughout the application: +1. Initialized in the Navbar on load if the user is logged in +2. Used by components to check user permissions +3. Updated when user profile changes + +```mermaid +flowchart TD + A["main.js"] --> B["createPinia()"] + B --> C["UserStore"] + C --> D["persistPlugin"] + + E["Navbar.vue"] --> |"fetchUser()"| F["API"] + F --> |"data"| G["userStore.initialize()"] + + C --> H["Components needing user data"] + C --> I["Authorization checks"] +``` + +Sources: +- [frontend/src/main.js:72-73]() - Pinia initialization with persistence plugin +- [frontend/src/components/navbar/Navbar.vue:371-377]() - User fetching and store initialization + +## 6. Security Considerations + +- JWT tokens are verified on the backend for authenticity +- Sessions expire after 7 days (cookie expiration) +- User state in localStorage has a 2-minute expiration as an additional safeguard +- Role checks are performed on both client and server sides +- Action limitations are implemented for users with incomplete profiles + +Sources: +- [internal/handlers/render/session.go:80-93]() - JWT verification process +- [frontend/src/packs/persistPinia.js:4-11]() - State expiration implementation + +## 7. Session Initialization Process + +The session initialization follows these steps: + +1. On application startup, the UserStore checks for existing saved state in localStorage +2. If the user has a valid `login_identity` cookie but the store is not initialized: + - The Navbar component makes an API call to fetch user data + - The UserStore is initialized with the fetched data +3. Protected routes and components check the UserStore to determine access rights + +Sources: +- [frontend/src/components/navbar/Navbar.vue:384-388]() - User data fetching on component mount +- [frontend/src/stores/UserStore.js:35-50]() - UserStore initialization logic \ No newline at end of file diff --git a/docs/development/12_User_Profile_and_Settings.md b/docs/development/12_User_Profile_and_Settings.md new file mode 100644 index 000000000..63537f3f0 --- /dev/null +++ b/docs/development/12_User_Profile_and_Settings.md @@ -0,0 +1,369 @@ +# User Profile and Settings + +This document provides a technical overview of the User Profile and Settings system in CSGHub. This system allows users to view and edit their profile information, manage account settings, and configure authentication options. + +For information about authentication mechanisms and session management, see [User Authentication and Session Management](#4.1). + +## 1. Component Architecture + +The User Profile and Settings system consists of several Vue components that handle different aspects of user profile management. + +### User Profile Component Structure + +```mermaid +graph TD + subgraph "Frontend Components" + ProfileVue["Profile.vue
User profile display"] + ProfileEditVue["ProfileEdit.vue
Edit profile form"] + MenuVue["Menu.vue
Settings navigation"] + UpdateUsernameVue["UpdateUsername.vue
First-time username prompt"] + end + + subgraph "State Management" + UserStoreJS["UserStore.js
Pinia store for user data"] + end + + subgraph "API Integration" + FetchApiUtil["useFetchApi
API client utility"] + UserAPI["User API Endpoints
/user/:username
/user/:uuid"] + JWTToken["JWT Token Management"] + end + + ProfileVue -->|"uses"| UserStoreJS + ProfileEditVue -->|"updates"| UserStoreJS + MenuVue -->|"reads"| UserStoreJS + UpdateUsernameVue -->|"updates"| UserStoreJS + + ProfileVue -->|"fetches other users"| FetchApiUtil + ProfileEditVue -->|"saves changes"| FetchApiUtil + + FetchApiUtil -->|"calls"| UserAPI + UserAPI -->|"refreshes"| JWTToken + JWTToken -->|"updates"| UserStoreJS +``` + +Sources: +- [frontend/src/components/user_settings/Profile.vue]() +- [frontend/src/components/user_settings/ProfileEdit.vue]() +- [frontend/src/components/user_settings/Menu.vue]() +- [frontend/src/components/popup/UpdateUsername.vue]() + +### Profile Display Component + +The `Profile.vue` component displays user profile information. Key features include: + +- Dynamic display based on whether viewing own profile or another user's profile +- Avatar and user information display +- Organization membership list +- Contact information display (when available) + +The component uses the `isCurrentUser` computed property to determine whether to load data from the `UserStore` (for current user) or fetch it from the API (for other users). + +```mermaid +graph TD + subgraph "Profile.vue" + ProfileView["Profile View"] + IsCurrentUser["isCurrentUser computed property"] + FetchUserInfo["fetchUserInfo() method"] + ProfileData["Profile Data Display"] + OrganizationSection["Organization Section"] + end + + subgraph "External Dependencies" + UserStoreJS["UserStore.js"] + UseFetchApi["useFetchApi utility"] + APIEndpoint["/user/:username API"] + end + + ProfileView -->|"initializes"| IsCurrentUser + IsCurrentUser -->|"true"| UserStoreJS + IsCurrentUser -->|"false"| FetchUserInfo + FetchUserInfo -->|"calls"| UseFetchApi + UseFetchApi -->|"requests"| APIEndpoint + + UserStoreJS -->|"provides data"| ProfileData + APIEndpoint -->|"provides data"| ProfileData + ProfileData -->|"renders"| OrganizationSection +``` + +Sources: +- [frontend/src/components/user_settings/Profile.vue:92-95]() - `isCurrentUser` computed property +- [frontend/src/components/user_settings/Profile.vue:100-114]() - `fetchUserInfo` method +- [frontend/src/components/user_settings/Profile.vue:43-66]() - Organization section + +## 2. Profile Editing System + +The profile editing functionality is implemented in `ProfileEdit.vue` and provides a form for users to update their profile information. + +### ProfileEdit Component Structure + +The `ProfileEdit.vue` component contains: + +- Avatar upload and removal functionality +- Username field (with special handling for first-time users) +- Email validation with regex pattern +- Form fields for phone, nickname, homepage, and bio +- Profile update API integration + +Profile updates follow this workflow: + +```mermaid +sequenceDiagram + participant User + participant ProfileEditVue as "ProfileEdit.vue" + participant ValidationSystem as "Form Validation" + participant UserStoreJS as "UserStore.js" + participant FetchApiUtil as "useFetchApi" + participant BackendAPI as "Backend API" + + User->>ProfileEditVue: Edit profile fields + ProfileEditVue->>ValidationSystem: Validate inputs (email, etc.) + ValidationSystem-->>ProfileEditVue: Validation result + + User->>ProfileEditVue: Click Save + + alt canChangeUsername is true + ProfileEditVue->>User: Show confirmation dialog + User->>ProfileEditVue: Confirm update + end + + ProfileEditVue->>FetchApiUtil: PUT /user/:uuid with profile data + FetchApiUtil->>BackendAPI: Send profile update + BackendAPI-->>FetchApiUtil: Return success response + + ProfileEditVue->>FetchApiUtil: PUT /internal_api/users/jwt_token + FetchApiUtil->>BackendAPI: Request new JWT + BackendAPI-->>FetchApiUtil: Return updated JWT + + FetchApiUtil->>UserStoreJS: Update store with new data + UserStoreJS-->>ProfileEditVue: Reflect updated data + ProfileEditVue->>User: Show success message +``` + +Sources: +- [frontend/src/components/user_settings/ProfileEdit.vue:1-316]() +- [frontend/src/components/user_settings/ProfileEdit.vue:248-292]() - `updateProfile` function +- [frontend/src/components/user_settings/ProfileEdit.vue:294-311]() - `saveProfile` function +- [frontend/src/components/user_settings/ProfileEdit.vue:163-172]() - Email validation + +### Avatar Management + +The `ProfileEdit.vue` component provides methods for avatar management: + +- `uploadImage()`: Opens file selection dialog +- `removeImage()`: Clears avatar image +- `uploadAvatar()`: Sends avatar to server using FormData + +The avatar is uploaded to `/internal_api/upload` endpoint using `csrfFetch` to include CSRF protection. + +Sources: +- [frontend/src/components/user_settings/ProfileEdit.vue:196-221]() - Avatar management methods + +## 3. Username Management + +CSGHub implements special handling for username updates, particularly for first-time users. + +### Username Update Workflow + +First-time users are prompted to set their username using the `UpdateUsername.vue` component, which displays as a modal dialog. The ability to change username is controlled through the `can_change_username` cookie. + +Key elements of username management: + +- Username validation including length, character restrictions, and pattern matching +- Special confirmation dialog for username changes +- Cookie-based tracking of username change capability +- JWT token refresh after username update + +The username validation includes multiple rules: +1. Required field validation +2. Length limits (2-64 characters) +3. Must start with a letter +4. Must end with a letter or number +5. Only allows letters, numbers, and specific special characters (-_.) +6. No consecutive special characters + +Sources: +- [frontend/src/components/popup/UpdateUsername.vue:1-141]() +- [frontend/src/components/popup/UpdateUsername.vue:55-96]() - Username validation rules +- [frontend/src/components/user_settings/ProfileEdit.vue:226-246]() - Username update confirmation + +## 4. Settings Navigation + +The `Menu.vue` component provides navigation within the User Settings section and displays user information. + +### Settings Menu Features + +- User avatar and name display at the top +- Navigation links to different settings sections +- Responsive design with tabs for mobile view +- Confirmation dialog for unsaved changes +- Conditional display of advanced options based on user permissions + +The menu offers different navigation options: +1. Profile settings (`/settings/profile`) +2. Access tokens (`/settings/access-token`) - only for non-limited users +3. SSH keys (`/settings/ssh-keys`) - only for non-limited users + +Sources: +- [frontend/src/components/user_settings/Menu.vue:1-151]() +- [frontend/src/components/user_settings/Menu.vue:31-46]() - Menu links +- [frontend/src/components/user_settings/Menu.vue:51-64]() - Mobile tabs +- [frontend/src/components/user_settings/Menu.vue:69-100]() - Confirmation dialog + +## 5. Integration with Other Systems + +The User Profile and Settings system integrates with several other components of CSGHub. + +### System Integration Map + +```mermaid +graph TD + subgraph "UserProfileSystem" + ProfileVue["Profile.vue"] + ProfileEditVue["ProfileEdit.vue"] + MenuVue["Menu.vue"] + end + + subgraph "OrganizationSystem" + OrgDetailVue["OrganizationDetail.vue"] + OrgSettingsVue["OrganizationSettings.vue"] + InviteMemberVue["InviteMember.vue"] + end + + subgraph "CollectionSystem" + AddToCollectionsVue["AddToCollections.vue"] + CollectionCardsVue["CollectionCards.vue"] + end + + subgraph "ResourceSystem" + ResourceConsoleVue["ResourceConsoleIndex.vue"] + EndpointsComponent["Endpoint Management"] + FinetuneComponent["Finetune Management"] + EvaluationTableVue["EvaluationTable.vue"] + end + + subgraph "RepositorySystem" + ProfileRepoListVue["ProfileRepoList.vue"] + RepoItemVue["RepoItem.vue"] + end + + ProfileVue -->|"displays"| OrgDetailVue + ProfileVue -->|"includes"| ProfileRepoListVue + ProfileRepoListVue -->|"shows"| RepoItemVue + + UserProfileSystem -->|"link to"| ResourceConsoleVue + ResourceConsoleVue -->|"manages"| EndpointsComponent + ResourceConsoleVue -->|"manages"| FinetuneComponent + ResourceConsoleVue -->|"manages"| EvaluationTableVue + + ProfileVue -->|"enables"| AddToCollectionsVue + AddToCollectionsVue -->|"displays"| CollectionCardsVue +``` + +Sources: +- [frontend/src/components/organizations/OrganizationDetail.vue]() +- [frontend/src/components/collections/AddToCollections.vue]() +- [frontend/src/components/resource_console/ResourceConsoleIndex.vue]() +- [frontend/src/components/shared/ProfileRepoList.vue]() + +### Organization Integration + +Users can belong to organizations, and the profile system displays organization membership: +- `Profile.vue` displays organization logos and links +- Organization role information (admin, write, read) affects available actions +- Organization details are retrieved via `/organization/:name` endpoint + +Sources: +- [frontend/src/components/user_settings/Profile.vue:43-66]() - Organization display section +- [frontend/src/components/organizations/OrganizationDetail.vue:66-68]() - Organization settings link + +### Collections Integration + +The profile system allows users to add repositories to their collections: +- Collections are displayed in the `ProfileRepoList.vue` component +- The `AddToCollections.vue` component enables adding repositories to collections +- Collection management is integrated with the user profile system via the UserStore + +Sources: +- [frontend/src/components/collections/AddToCollections.vue:1-190]() +- [frontend/src/components/shared/ProfileRepoList.vue:17-27]() - Collections section + +### Resource Console Integration + +The user profile links to the Resource Console for managing AI resources: +- Endpoints (model deployment) +- Finetunes (model training) +- Evaluations (model testing) + +Sources: +- [frontend/src/components/resource_console/ResourceConsoleIndex.vue:1-174]() +- [frontend/src/components/resource_console/EvaluationTable.vue:1-336]() + +## 6. Internationalization + +The User Profile and Settings system supports internationalization through Vue i18n: + +### Profile Localization + +- User interface text is defined in locale files +- Text elements use the `$t()` function for translations +- English and Chinese translations are provided +- Localization covers all user-facing text in profile components + +Key localization files: +- English: [frontend/src/locales/en_js/profile.js]() +- Chinese: [frontend/src/locales/zh_js/profile.js]() + +Example of localized elements: +- Account settings title +- Form field labels +- Button text +- Validation messages +- Confirmation dialogs + +Sources: +- [frontend/src/locales/en_js/profile.js:1-35]() +- [frontend/src/locales/zh_js/profile.js:1-35]() +- [frontend/src/locales/en_js/navbar.js:8-9]() - Profile navigation items +- [frontend/src/locales/zh_js/navbar.js:8-9]() - Profile navigation items (Chinese) + +## 7. UI Components and Data Flow + +The User Profile and Settings system uses Element Plus UI components and follows a structured data flow pattern. + +### Key UI Components + +- `el-avatar`: Displays user profile image +- `el-input`: Form input fields +- `el-form` and `el-form-item`: Form structure with validation +- `el-dialog`: Modal dialogs +- `el-tooltip`: Information tooltips +- `el-tabs`: Mobile navigation + +### Data Flow Pattern + +1. Component initialization + - Load data from UserStore (for current user) + - Fetch data from API (for other users) + +2. Form interaction + - Two-way binding with v-model + - Input validation on change + - Track unsaved changes + +3. Data submission + - Validate form data + - Submit to API endpoint + - Update UserStore + - Show success/error notifications + +4. JWT Token Refresh + - After successful profile update + - Maintains session with updated information + +Sources: +- [frontend/src/components/user_settings/ProfileEdit.vue:31-105]() - Form fields +- [frontend/src/components/user_settings/ProfileEdit.vue:248-292]() - Data submission +- [frontend/src/components/user_settings/Profile.vue:4-14]() - Profile display + +The User Profile and Settings system provides a comprehensive interface for users to manage their personal information and account settings while integrating seamlessly with other components of the CSGHub platform. \ No newline at end of file diff --git a/docs/development/13_Organizations_and_Collections.md b/docs/development/13_Organizations_and_Collections.md new file mode 100644 index 000000000..f396c1c78 --- /dev/null +++ b/docs/development/13_Organizations_and_Collections.md @@ -0,0 +1,272 @@ +# Organizations and Collections + +This document explains the organization and collection management features of CSGHub. Organizations provide a way to group users with shared access to repositories, while collections enable users to organize repositories into custom groups. For information about user authentication and profile management, see [User Authentication and Session Management](#4.1) and [User Profile and Settings](#4.2). + +## 1. Organizations + +Organizations in CSGHub represent teams, companies, or communities that collaborate on repositories. They enable centralized management of members, repositories, and permissions. + +```mermaid +graph TD + subgraph "Organization" + OrgProps["Organization Properties"] + OrgProps --> Path["path"] + OrgProps --> Name["name/nickname"] + OrgProps --> Logo["logo"] + OrgProps --> Homepage["homepage"] + OrgProps --> OrgType["org_type"] + OrgProps --> Verified["verified status"] + + OrgMembership["Member Management"] + OrgMembership --> AdminRole["admin role"] + OrgMembership --> WriteRole["write role"] + OrgMembership --> ReadRole["read role"] + end + + subgraph "Related Entities" + User["User"] + Repository["Repository"] + end + + OrgMembership -- "manages access" --> Repository + User -- "belongs to" --> OrgMembership + Organization -- "owns" --> Repository +``` + +Sources: [frontend/src/components/organizations/OrganizationDetail.vue:6-14](), [frontend/src/components/organizations/OrganizationSettings.vue:46-53]() + +### 1.1 Organization Structure and Properties + +Organizations have several key properties: + +| Property | Description | +|----------|-------------| +| path | Unique identifier used in URLs (e.g., `/organizations/{path}`) | +| name | Display name shown in the interface | +| nickname | Alternative display name | +| logo | Organization's avatar image | +| homepage | External website URL | +| org_type | Classification of organization (e.g., "社区组织" - Community Organization) | +| verified | Boolean indicating official verification status | + +Organization data is fetched from the backend API endpoint `/organization/{name}` which returns the organization details. + +Sources: [frontend/src/components/organizations/OrganizationDetail.vue:132-143](), [frontend/src/components/organizations/OrganizationSettings.vue:57-71]() + +### 1.2 Organization Membership and Roles + +Organizations have a hierarchical role system that controls access levels: + +```mermaid +graph TD + subgraph "User Roles in Organization" + AdminRole["admin role"] + WriteRole["write role"] + ReadRole["read role"] + end + + subgraph "Permissions" + CreateRepo["Create repositories"] + EditOrgSettings["Edit organization settings"] + ManageMembers["Manage members"] + AccessRepos["Access repositories"] + end + + AdminRole --> CreateRepo + AdminRole --> EditOrgSettings + AdminRole --> ManageMembers + AdminRole --> AccessRepos + + WriteRole --> CreateRepo + WriteRole --> AccessRepos + + ReadRole --> AccessRepos +``` + +Sources: [frontend/src/components/organizations/OrganizationDetail.vue:21-68](), [frontend/src/components/organizations/OrganizationDetail.vue:156-165]() + +Users can have one of the following roles in an organization: + +1. **admin** - Can manage organization settings, members, and create repositories +2. **write** - Can create repositories and access organization content +3. **read** - Can access organization content but cannot create repositories + +The current user's role in an organization is determined by calling the API endpoint `/organization/{name}/members/{username}`. + +Organization members are displayed in the organization detail view with their roles. The organization admin can invite new members through the `InviteMember` component. + +Sources: [frontend/src/components/organizations/OrganizationDetail.vue:81-97](), [frontend/src/components/organizations/OrganizationDetail.vue:146-155]() + +### 1.3 Organization Repository Management + +Organizations can own repositories of various types: + +```mermaid +graph TD + subgraph "Organization" + OrgRepo["Organization Repositories"] + end + + subgraph "Repository Types" + Models["Models Repository"] + Datasets["Datasets Repository"] + Spaces["Spaces Repository"] + Codes["Codes Repository"] + end + + OrgRepo --> Models + OrgRepo --> Datasets + OrgRepo --> Spaces + OrgRepo --> Codes + + subgraph "Creation Routes" + ModelCreate["/models/new?orgName={name}"] + DatasetCreate["/datasets/new?orgName={name}"] + SpaceCreate["/spaces/new?orgName={name}"] + CodeCreate["/codes/new?orgName={name}"] + end + + Models --- ModelCreate + Datasets --- DatasetCreate + Spaces --- SpaceCreate + Codes --- CodeCreate +``` + +Sources: [frontend/src/components/organizations/OrganizationDetail.vue:27-63]() + +Users with admin or write permissions can create repositories under the organization. The repository creation UI includes options to specify the organization as the owner. When creating a repository through the organization interface, the organization name is automatically passed as a URL parameter. + +## 2. Collections + +Collections provide a way for users to group related repositories together, regardless of ownership. Unlike organizations which primarily focus on user grouping and permissions, collections focus on repository grouping for organizational purposes. + +```mermaid +graph TD + subgraph "Collections System" + Collection["Collection"] + Collection --> CollID["id"] + Collection --> CollName["name"] + Collection --> Repositories["repositories[]"] + + User["User"] -- "owns" --> Collection + Repository["Repository"] -- "belongs to" --> Collection + end + + subgraph "API Endpoints" + ListCollections["/user/{username}/collections"] + AddToCollection["/collections/{id}/repos"] + end + + User --- ListCollections + Collection --- AddToCollection +``` + +Sources: [frontend/src/components/collections/AddToCollections.vue:97-106](), [frontend/src/components/collections/AddToCollections.vue:129-146]() + +### 2.1 Collection Management + +Collections are user-specific organizational tools. Each user can create and manage their own collections, which are accessible via the API endpoint `/user/{username}/collections`. + +The key operations for collections include: + +1. **Viewing collections**: Fetching collections for a user +2. **Adding repositories to collections**: Adding one or more repositories to a collection + +Sources: [frontend/src/components/collections/AddToCollections.vue:97-106]() + +### 2.2 Adding Repositories to Collections + +The "Add to Collection" workflow is implemented in the `AddToCollections` component: + +```mermaid +flowchart TD + Start["User clicks 'Add Collection' button"] --> CheckLogin{Is user logged in?} + CheckLogin -- No --> Redirect["Redirect to login page"] + CheckLogin -- Yes --> OpenDialog["Open collection dialog"] + OpenDialog --> SelectCollection["User selects collection"] + SelectCollection --> SubmitForm["User confirms selection"] + SubmitForm --> ValidateForm{Is selection valid?} + ValidateForm -- No --> ShowError["Show warning message"] + ValidateForm -- Yes --> SendRequest["Send API request to add repository"] + SendRequest --> CheckResponse{Success?} + CheckResponse -- Yes --> ShowSuccess["Show success message"] + CheckResponse -- No --> ShowError2["Show error message"] + ShowSuccess --> CloseDialog["Close dialog"] +``` + +Sources: [frontend/src/components/collections/AddToCollections.vue:108-146]() + +The process of adding a repository to a collection involves: + +1. Opening the "Add to Collection" dialog +2. Fetching the user's collections +3. Selecting a collection +4. Sending a request to the API endpoint `/collections/{id}/repos` with the repository ID +5. Displaying a success message on completion + +The data structure for adding a repository to a collection is: + +```json +{ + "repo_ids": ["repository-id"] +} +``` + +Sources: [frontend/src/components/collections/AddToCollections.vue:130-135]() + +### 2.3 Collection Integration + +Collections are integrated with repository views. Users can add repositories to collections directly from repository interfaces through the `AddToCollections` component. This component displays a button that opens a dialog for selecting a collection. + +Collections are displayed in user profiles, allowing users to quickly access their organized repository groups. + +Sources: [frontend/src/components/collections/AddToCollections.vue:1-5]() + +## 3. Integration with User Profiles + +Both organizations and collections are displayed in user profiles, showing the user's affiliations and organizational structure. + +```mermaid +graph TD + subgraph "User Profile" + UserInfo["User Information"] + UserOrgs["User's Organizations"] + UserColls["User's Collections"] + end + + subgraph "API Endpoints" + ProfileEndpoint["/user/{username}"] + OrgsEndpoint["/user/{username}/collections"] + end + + UserInfo -- "displays" --> UserDetails["Name, Avatar, Bio, etc."] + UserOrgs -- "displays" --> OrgList["List of Organization Memberships"] + OrgList --> OrgLogo["Organization Logo"] + OrgList --> OrgName["Organization Name"] + + ProfileEndpoint --> UserInfo + ProfileEndpoint --> UserOrgs + OrgsEndpoint --> UserColls +``` + +Sources: [frontend/src/components/user_settings/Profile.vue:43-66]() + +The user profile displays organization memberships, showing organization logos with tooltips containing organization names. When a user views their own profile, the organization data is retrieved from the user store, while for other users' profiles, it's fetched from the API endpoint `/user/{username}`. + +Sources: [frontend/src/components/user_settings/Profile.vue:100-114]() + +## 4. Summary + +Organizations and collections provide two complementary approaches to structure in CSGHub: + +| Feature | Organizations | Collections | +|---------|--------------|------------| +| **Primary purpose** | Team collaboration | Repository organization | +| **Permission control** | Yes - role-based | No - personal only | +| **Repository ownership** | Yes | No | +| **Member management** | Yes | No | +| **User affiliation** | Yes | No | +| **Custom grouping** | No | Yes | +| **Visibility in profile** | Yes | Yes | + +Organizations focus on team collaboration with hierarchical permissions, while collections provide personal organization of repositories across different owners and organizations. Together, they provide comprehensive tools for managing repository access and organization in CSGHub. \ No newline at end of file diff --git a/docs/development/14_Frontend_Architecture.md b/docs/development/14_Frontend_Architecture.md new file mode 100644 index 000000000..3db9754c5 --- /dev/null +++ b/docs/development/14_Frontend_Architecture.md @@ -0,0 +1,413 @@ +# Frontend Architecture + +This document provides an overview of the frontend architecture for CSGHub, explaining the core structure, key components, and design patterns used throughout the client-side application. For information about specific components like repositories or endpoints, see [Repository Management System](#2) or [Model Deployment and Fine-tuning](#3). + +## Overview + +CSGHub's frontend is built as a single-page application (SPA) using Vue.js 3, leveraging modern frontend technologies and architectural patterns. The application provides interfaces for managing LLM assets, including repositories, models, datasets, endpoints, and evaluations. + +```mermaid +graph TB + subgraph "Frontend Core Structure" + VueApp["Vue Application"] + Router["Vue Router"] + Pinia["Pinia State Management"] + I18n["Internationalization"] + ElementPlus["ElementPlus UI Library"] + + VueApp --> Router + VueApp --> Pinia + VueApp --> I18n + VueApp --> ElementPlus + end + + subgraph "Entry Points" + MainJS["main.js"] + AdminJS["admin.js"] + + MainJS --> VueApp + AdminJS --> VueApp + end + + subgraph "API Integration" + FetchAPI["useFetchApi"] + Auth["Authentication"] + BackendAPI["Backend API Services"] + + FetchAPI --> BackendAPI + Auth --> BackendAPI + end +``` + +Sources: [frontend/src/main.js:1-167](), [frontend/src/admin.js:1-70](), [frontend/package.json:1-55]() + +## Core Technologies + +The frontend application is built using the following key technologies: + +| Technology | Purpose | Implementation | +|------------|---------|----------------| +| Vue.js 3 | Core framework | Composition API and Options API | +| Vue Router | Page navigation | Route configuration in main.js | +| Pinia | State management | Store pattern with persistence | +| Element Plus | UI component library | Styled components with custom theming | +| i18n | Internationalization | English and Chinese language support | +| Vite | Build tool | Development and production builds | + +Sources: [frontend/package.json:14-42](), [frontend/src/main.js:1-9]() + +## Application Architecture + +### Entry Points and Initialization + +The application has two main entry points: + +1. **Main Application**: Initializes the primary user interface +2. **Admin Application**: Initializes the administration panel + +```mermaid +graph TD + MainJS["main.js"] --> CreateApp["createApp()"] + AdminJS["admin.js"] --> CreateAdminApp["createApp(AdminLayout)"] + + CreateApp --> RegisterComponents["Register Global Components"] + CreateApp --> SetupRouter["Setup Vue Router"] + CreateApp --> ConfigurePinia["Configure Pinia"] + CreateApp --> SetupI18n["Setup i18n"] + CreateApp --> MountApp["Mount App (#app)"] + + CreateAdminApp --> RegisterAdminComponents["Register Admin Components"] + CreateAdminApp --> SetupAdminRouter["Setup Admin Router"] + CreateAdminApp --> ConfigureAdminPinia["Configure Admin Pinia"] + CreateAdminApp --> SetupAdminI18n["Setup Admin i18n"] + CreateAdminApp --> MountAdminApp["Mount Admin App (#admin)"] +``` + +Sources: [frontend/src/main.js:75-167](), [frontend/src/admin.js:46-70]() + +### Component Structure + +The application follows a hierarchical component structure organized by feature domains: + +```mermaid +graph TD + RootApp["App Root"] + + RootApp --> Navbar["Navbar Component"] + RootApp --> Router["Router View"] + + Router --> RepoSystem["Repository Components"] + Router --> UserSystem["User Components"] + Router --> EndpointSystem["Endpoint Components"] + Router --> OrgSystem["Organization Components"] + Router --> AdminSystem["Admin Components"] + + RepoSystem --> RepoDetail["RepoDetail"] + RepoSystem --> RepoCards["RepoCards"] + + UserSystem --> Profile["Profile"] + UserSystem --> Settings["Settings"] + + EndpointSystem --> EndpointDetail["EndpointDetail"] + EndpointSystem --> NewEndpoint["NewEndpoint"] + + OrgSystem --> OrganizationDetail["OrganizationDetail"] + OrgSystem --> NewOrganization["NewOrganization"] + + AdminSystem --> AdminDashboard["AdminDashboard"] + AdminSystem --> AdminUserList["AdminUserList"] +``` + +Sources: [frontend/src/main.js:16-69](), [frontend/src/main.js:146-158]() + +## State Management + +The application uses Pinia for state management with a custom persistence layer for maintaining state between sessions. + +### UserStore + +The UserStore is a central store that manages user authentication state and profile information: + +```mermaid +graph LR + UserStore["UserStore"] + + subgraph "State Properties" + UserInfo["User Information
(username, email, etc.)"] + AuthState["Authentication State"] + Permissions["User Permissions"] + end + + subgraph "Actions" + Initialize["initialize()"] + ClearStore["clearStore()"] + end + + subgraph "Computed Properties" + IsLoggedIn["isLoggedIn"] + IsAdmin["isAdmin"] + ActionLimited["actionLimited"] + end + + UserStore --> UserInfo + UserStore --> AuthState + UserStore --> Permissions + UserStore --> Initialize + UserStore --> ClearStore + UserStore --> IsLoggedIn + UserStore --> IsAdmin + UserStore --> ActionLimited +``` + +Sources: [frontend/src/stores/UserStore.js:1-95](), [frontend/src/packs/persistPinia.js:1-20]() + +### State Persistence + +A custom persistence plugin ensures that state is preserved across page reloads, with automatic expiration: + +1. State is stored in localStorage with a timestamp +2. On initialization, checks if stored state is still valid (within expiration window) +3. Subscribes to store changes and updates localStorage + +Sources: [frontend/src/packs/persistPinia.js:1-20]() + +## Routing and Navigation + +The application uses Vue Router to manage navigation between different views: + +```mermaid +graph TD + Router["Vue Router"] + + subgraph "Main Routes" + Home["/"] + RepoDetail["/username/repo-name"] + Profile["/profile/:username"] + Settings["/settings/*"] + ResourceConsole["/resource-console"] + AdminPanel["/admin_panel/*"] + end + + subgraph "Creation Routes" + NewModel["/models/new"] + NewDataset["/datasets/new"] + NewCode["/codes/new"] + NewSpace["/spaces/new"] + NewEndpoint["/endpoints/new"] + NewOrg["/organizations/new"] + end + + Router --> Home + Router --> RepoDetail + Router --> Profile + Router --> Settings + Router --> ResourceConsole + Router --> AdminPanel + + Router --> NewModel + Router --> NewDataset + Router --> NewCode + Router --> NewSpace + Router --> NewEndpoint + Router --> NewOrg +``` + +Sources: [frontend/src/main.js:146-158]() + +## UI Component System + +The application leverages Element Plus as the primary UI component library, with custom styling and additional components: + +```mermaid +graph TD + ElementPlus["Element Plus"] + + subgraph "Base Components" + ElButton["ElButton"] + ElInput["ElInput"] + ElTable["ElTable"] + ElForm["ElForm"] + ElMenu["ElMenu"] + ElDropdown["ElDropdown"] + end + + subgraph "Custom Components" + SvgIcon["SvgIcon"] + CsgButton["CsgButton"] + FlashMessage["FlashMessage"] + RepoCards["RepoCards"] + end + + ElementPlus --> ElButton + ElementPlus --> ElInput + ElementPlus --> ElTable + ElementPlus --> ElForm + ElementPlus --> ElMenu + ElementPlus --> ElDropdown + + CustomComponents --> SvgIcon + CustomComponents --> CsgButton + CustomComponents --> FlashMessage + CustomComponents --> RepoCards +``` + +Sources: [frontend/src/main.js:160-162](), [frontend/src/admin.js:51-52]() + +## Internationalization (i18n) + +The application supports multiple languages through Vue i18n: + +1. Default language detection based on browser settings +2. User-selectable language preference stored in cookies +3. Translation files for English and Chinese + +```mermaid +graph LR + I18n["Vue I18n"] + + Locales["Locales"] --> English["English (en)"] + Locales --> Chinese["Chinese (zh)"] + + I18n --> Locales + I18n --> LocaleDetection["Locale Detection"] + I18n --> LocaleStorage["Locale Storage (Cookies)"] + + Components["UI Components"] --> TranslatedText["$t('key')"] + TranslatedText --> I18n +``` + +Sources: [frontend/src/main.js:132-139](), [frontend/src/admin.js:29-36]() + +## Build and Development System + +The application uses Vite for building and serving the application in both development and production environments: + +```mermaid +graph TB + Vite["Vite Build Tool"] + + subgraph "Development Config" + DevServer["Dev Server"] + HotReload["Hot Module Reload"] + SourceMaps["Source Maps"] + end + + subgraph "Production Config" + Optimization["Code Optimization"] + Minification["Terser Minification"] + ChunkSplitting["Chunk Splitting"] + end + + Vite --> DevConfig["Development Configuration"] + Vite --> ProdConfig["Production Configuration"] + + DevConfig --> DevServer + DevConfig --> HotReload + DevConfig --> SourceMaps + + ProdConfig --> Optimization + ProdConfig --> Minification + ProdConfig --> ChunkSplitting +``` + +Sources: [frontend/vite.config.js:32-118]() + +### Build Configuration + +The build system is configured to optimize the application differently for development and production: + +1. **Development**: Optimized for fast rebuilds and debugging + - No minification + - Source maps enabled + - Treeshake disabled + +2. **Production**: Optimized for performance and size + - Terser minification + - Console logs removed + - Optimized chunk splitting + +Sources: [frontend/vite.config.js:32-76](), [frontend/vite.config.js:78-118]() + +## API Integration + +The frontend integrates with backend services through a centralized API utility: + +```mermaid +graph TD + Components["Vue Components"] + + Components --> UseFetchApi["useFetchApi()"] + + UseFetchApi --> FetchOptions["Configure Fetch Options"] + UseFetchApi --> HandleResponse["Handle Response"] + UseFetchApi --> HandleErrors["Handle Errors"] + + FetchOptions --> API["Backend API"] + + API --> JsonResponse["JSON Response"] + JsonResponse --> HandleResponse +``` + +Sources: [frontend/src/main.js:324](), [frontend/src/navbar/Navbar.vue:372]() + +## Authentication Flow + +The authentication system integrates with the backend through JWT tokens and cookies: + +```mermaid +graph TB + Login["Login Page"] + + Login --> AuthProvider["Authentication Provider"] + AuthProvider --> JWT["JWT Token"] + + JWT --> Cookies["Store in Cookies"] + Cookies --> UserStore["Update UserStore"] + + UserStore --> IsLoggedIn["Update isLoggedIn"] + UserStore --> FetchUserInfo["Fetch User Info"] + + FetchUserInfo --> UserProfile["User Profile Data"] + UserProfile --> UserStore + + Logout["Logout Action"] --> ClearCookies["Clear Cookies"] + ClearCookies --> ClearUserStore["Clear UserStore"] +``` + +Sources: [frontend/src/navbar/Navbar.vue:379-382](), [frontend/src/stores/UserStore.js:25-33]() + +## Responsive Design + +The frontend implements responsive design through Tailwind CSS and media queries: + +1. Page layout adapts to different screen sizes +2. Mobile-specific navigation for smaller screens +3. Responsive components that adjust to container width + +Sources: [frontend/src/navbar/Navbar.vue:3-255](), [frontend/package.json:51]() + +## Performance Optimizations + +The application includes several performance optimizations: + +1. **Code Splitting**: Separates vendor code from application code +2. **Lazy Loading**: Loads components only when needed +3. **Chunk Optimization**: Organizes code into logical chunks +4. **CSS Optimization**: Uses Tailwind CSS for efficient styling + +Sources: [frontend/vite.config.js:83-101]() + +## Security Considerations + +The frontend implements several security best practices: + +1. JWT token-based authentication +2. Session expiration +3. Protected routes requiring authentication +4. Input validation for user inputs + +Sources: [frontend/src/stores/UserStore.js:51-58]() + +## Conclusion + +The CSGHub frontend architecture follows modern best practices for Vue.js applications, with a component-based structure, efficient state management, and robust internationalization. The application is built to be maintainable, scalable, and performant while providing a rich user interface for managing machine learning assets. \ No newline at end of file diff --git a/docs/development/15_Component_System_and_Navigation.md b/docs/development/15_Component_System_and_Navigation.md new file mode 100644 index 000000000..a9966ab7f --- /dev/null +++ b/docs/development/15_Component_System_and_Navigation.md @@ -0,0 +1,491 @@ +# Component System and Navigation + +This document details the component architecture and navigation system of the CSGHub platform. It explains how the frontend components are organized, how the navigation system works across different screen sizes, and how user state affects navigation options. For information about internationalization specifically, see the [Internationalization (i18n)](#5.2) page. + +## 1. Frontend Component Architecture + +The CSGHub frontend is built using Vue 3 with the Composition API, using a modular component architecture that separates concerns by feature areas. + +### 1.1 Core Component Structure + +```mermaid +graph TD + subgraph "Application Entry" + A["main.js"] --> B["Vue App Instance"] + end + + subgraph "Core Components" + B --> N["Navbar"] + B --> C["Common Components"] + B --> F["Feature Components"] + B --> S["Shared Components"] + end + + subgraph "Navigation System" + N --> NM["MenuItems"] + N --> ND["Dropdown Menus"] + N --> NR["Responsive Logic"] + end + + subgraph "Common Components" + C --> C1["SvgIcon"] + C --> C2["CsgButton"] + C --> C3["FlashMessage"] + end + + subgraph "Feature Components" + F --> FR["Repository Components"] + F --> FU["User Components"] + F --> FE["Endpoint Components"] + F --> FC["Collection Components"] + end + + subgraph "Shared Components" + S --> S1["RepoDetail"] + S --> S2["ProfileRepoList"] + S --> S3["RepoCards"] + end + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class A,N,NM,C1,C2,C3 primary; +``` + +Sources: [frontend/src/main.js:1-167](), [frontend/src/components/navbar/Navbar.vue:1-443]() + +### 1.2 Component Registration and Initialization + +The main application initializes the component system through global registration of core components and setting up Vue plugins: + +1. **Global Component Registration**: Frequently used components like `SvgIcon` and `CsgButton` are registered globally +2. **Plugin Integration**: Element Plus UI library, Vue Router, i18n, and Pinia state management +3. **Provider Values**: Global configuration values provided to all components + +```mermaid +sequenceDiagram + participant main as "main.js" + participant app as "Vue App" + participant plugins as "Plugins" + participant components as "Components" + + main->>app: Create app instance + main->>plugins: Configure Pinia + main->>plugins: Configure i18n + main->>plugins: Register Element Plus + main->>components: Register global components + app->>components: Mount components + main->>app: Provide global values + main->>app: Mount to #app element +``` + +Sources: [frontend/src/main.js:75-167]() + +## 2. Navigation System + +The navigation system provides the primary user interface for moving between different sections of the application. It adapts to both desktop and mobile screen sizes. + +### 2.1 Navigation Components Hierarchy + +```mermaid +graph TD + subgraph "Navigation Components" + N["Navbar.vue"] --> NM["MenuItems.vue"] + N --> NB["Broadcast.vue"] + N --> NU["UpdateUsername.vue"] + end + + subgraph "Navigation UI Elements" + NM --> NMI["Menu Items"] + NM --> NMS["Submenu Items"] + N --> ND["User Dropdown"] + N --> NL["Language Dropdown"] + N --> NM["Mobile Menu"] + end + + subgraph "State Integration" + N --> US["UserStore"] + US --> UA["Authentication State"] + US --> UP["User Profile"] + US --> UR["User Roles"] + end + + subgraph "Responsive Behavior" + N --> RB["Screen Resize Handler"] + RB --> RL["Layout Adaptation"] + RL --> RD["Desktop Layout"] + RL --> RM["Mobile Layout"] + end + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class N,NM,US primary; +``` + +Sources: [frontend/src/components/navbar/Navbar.vue:1-443](), [frontend/src/components/navbar/MenuItems.vue:1-236]() + +### 2.2 Menu Structure and Generation + +The navigation menu items are dynamically generated and organized based on screen size. On smaller screens, some items are moved to a "More" dropdown menu. + +```mermaid +graph LR + subgraph "Menu Generation Process" + MI["Raw Menu Items"] --> RS["Resize Handler"] + RS --> SW["Screen Width Check"] + + SW -->|"> 1280px"| FM["Full Menu"] + SW -->|"1024-1280px"| RM["Partial Menu + More Dropdown"] + SW -->|"768-1024px"| MM["Minimal Menu + More Dropdown"] + SW -->|"< 768px"| MB["Mobile Menu"] + end + + subgraph "Menu Item Categories" + MC1["Models"] + MC2["Datasets"] + MC3["Spaces"] + MC4["Codes"] + MC5["Collections"] + MC6["Prompts Library"] + end + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class MI,RS,SW primary; +``` + +Sources: [frontend/src/components/navbar/MenuItems.vue:128-211]() + +## 3. User State Integration with Navigation + +The navigation system adapts based on the user's authentication state and profile. The Pinia store manages this state and provides it to the navigation components. + +### 3.1 User Store Structure + +```mermaid +graph TD + subgraph "UserStore" + US["useUserStore()"] + US --> UP["User Profile State"] + US --> UA["Authentication State"] + US --> UR["User Roles"] + US --> UI["User Interface Restrictions"] + end + + subgraph "User Profile State" + UP --> UP1["username"] + UP --> UP2["nickname"] + UP --> UP3["email"] + UP --> UP4["avatar"] + UP --> UP5["uuid"] + end + + subgraph "Authentication State" + UA --> UA1["isLoggedIn"] + UA --> UA2["hasEmail"] + UA --> UA3["canChangeUsername"] + end + + subgraph "User Roles" + UR --> UR1["isAdmin"] + UR --> UR2["isSuperUser"] + end + + subgraph "User Interface Restrictions" + UI --> UI1["actionLimited"] + end + + subgraph "UserStore Methods" + UM1["initialize()"] + UM2["clearStore()"] + UM3["refreshCanChangeUsernameCookie()"] + end + + US --> UM1 + US --> UM2 + US --> UM3 + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class US,UA1,UR1,UI1 primary; +``` + +Sources: [frontend/src/stores/UserStore.js:1-95]() + +### 3.2 Navigation State Flow + +The navigation system uses the user state to conditionally render different menu options and UI elements: + +```mermaid +sequenceDiagram + participant N as "Navbar.vue" + participant US as "UserStore" + participant API as "Backend API" + participant C as "Cookies" + + Note over N,US: Component Initialization + N->>C: Get login_identity cookie + alt Cookie exists + N->>US: Check if store initialized + alt Store not initialized + N->>API: fetchUser() + API-->>N: User data + N->>US: initialize(userData) + US-->>N: Update computed properties + end + else No cookie + N->>N: Show login/register buttons + end + + Note over N,US: Computed Properties + US-->>N: isLoggedIn + US-->>N: isAdmin + US-->>N: hasEmail + US-->>N: canChangeUsername + US-->>N: actionLimited + + Note over N: Conditional Rendering + alt isLoggedIn + N->>N: Show user dropdown + alt isAdmin && !actionLimited + N->>N: Show admin panel link + end + alt !actionLimited + N->>N: Show resource console link + end + else Not logged in + N->>N: Show login/register buttons + end + + alt initialized && isLoggedIn && conditions + N->>N: Show profile warnings + end +``` + +Sources: [frontend/src/components/navbar/Navbar.vue:60-315](), [frontend/src/stores/UserStore.js:25-33]() + +## 4. Responsive Design and Styling + +The CSGHub navigation system is designed to be responsive across different device sizes, with distinct layouts for desktop and mobile views. + +### 4.1 Responsive Breakpoints + +The application uses Tailwind CSS with custom breakpoints to manage responsive layouts: + +| Breakpoint | Screen Width | Description | +|------------|--------------|-------------| +| 3xl | ≥ 1536px | Extra large desktop | +| 2xl | < 1536px | Large desktop | +| xl | < 1280px | Medium desktop | +| lg | < 1024px | Small desktop/Tablet landscape | +| md | < 768px | Tablet portrait | +| sm | < 640px | Mobile | + +Sources: [frontend/tailwind.config.js:115-122]() + +### 4.2 Navigation Layout Changes + +The navigation system adapts its layout based on screen size: + +```mermaid +graph TD + subgraph "Responsive Navigation Behavior" + RS["Resize Event Handler"] --> SW["Screen Width Detection"] + + SW -->|"> 1280px"| F["Full Desktop Layout
All menu items visible"] + SW -->|"1024-1280px"| P1["Partial Desktop Layout
5 items visible, rest in More menu"] + SW -->|"768-1024px"| P2["Compact Desktop Layout
3 items visible, rest in More menu"] + SW -->|"< 768px"| M["Mobile Layout
Hamburger menu with drawer"] + end + + subgraph "Mobile Layout Components" + M --> MD["El-Drawer Component"] + MD --> MM["Mobile Menu"] + MM --> ML["Vertical Menu Items"] + end + + subgraph "Screen Width Logic" + WL["window.innerWidth"] --> RI["initMenuItems()"] + RI --> SA["Slice Array Based on Width"] + SA --> AM["Assign to allNavItems"] + SA --> MO["Assign to moreItems"] + end + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class RS,SW,M primary; +``` + +Sources: [frontend/src/components/navbar/MenuItems.vue:172-210](), [frontend/src/components/navbar/Navbar.vue:258-285]() + +## 5. Styling System and UI Components + +The navigation system utilizes a combination of Element Plus UI components and custom styling with Tailwind CSS. + +### 5.1 UI Component Theming + +The application implements a comprehensive design system with custom CSS variables for colors, shadows, and border radii that are applied to Element Plus components. + +```mermaid +graph TD + subgraph "CSS Variable Hierarchy" + V["_variables.css"] --> C["Color Variables"] + V --> S["Shadow Variables"] + V --> B["Border Radius Variables"] + V --> EP["Element Plus Overrides"] + end + + subgraph "Button System" + BC["Button CSS"] --> BT["Button Types"] + BC --> BS["Button Sizes"] + BC --> BV["Button Variants"] + end + + subgraph "Menu Style Customization" + MC["Menu CSS"] --> MH["Horizontal Menu"] + MC --> MV["Vertical Menu"] + MC --> MS["Submenu Styles"] + end + + subgraph "Application Styles" + AC["application.css"] --> V + AC --> BC + AC --> MC + AC --> IC["Input CSS"] + AC --> UC["Upload CSS"] + end + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class V,BC,MC primary; +``` + +Sources: [frontend/src/assets/stylesheets/element-plus/_variables.css:1-94](), [frontend/src/assets/stylesheets/element-plus/button.css:1-449](), [frontend/src/assets/stylesheets/element-plus/menu.css:1-36]() + +### 5.2 Responsive Width Utility + +The application uses a custom utility class for responsive container widths: + +```css +.page-responsive-width { + @apply 3xl:w-[1536px] 2xl:w-[1280px] xl:w-[1024px] lg:w-[768px] md:w-[640px] sm:w-full mx-auto; +} +``` + +This class is used in the navigation component to ensure proper width at different screen sizes: + +```html +
+``` + +Sources: [frontend/src/style.css:135-137](), [frontend/src/components/navbar/Navbar.vue:4-5]() + +## 6. State Persistence + +The navigation system relies on persistent user state for displaying the correct UI options. This is implemented using the Pinia store with a custom persistence plugin. + +### 6.1 State Persistence Mechanism + +```mermaid +graph TD + subgraph "State Persistence Flow" + P["Pinia Store"] --> PP["Persist Plugin"] + PP --> LC["Load from LocalStorage"] + + LC --> TC["Check Timestamp"] + TC -->|"< 2 mins old"| RP["Restore State"] + TC -->|"≥ 2 mins old"| CP["Clear Persisted State"] + + P --> S["State Changes"] + S --> SP["Save to LocalStorage"] + SP --> AT["Add Timestamp"] + end + + subgraph "Persisted User Properties" + UP["User Properties"] + UP --> U1["username"] + UP --> U2["email"] + UP --> U3["avatar"] + UP --> U4["roles"] + UP --> U5["uuid"] + end + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class P,PP,TC primary; +``` + +Sources: [frontend/src/packs/persistPinia.js:1-20](), [frontend/src/stores/UserStore.js:89-93]() + +## 7. Admin Navigation Integration + +The CSGHub platform includes a separate administration interface with its own navigation system that integrates with the main component architecture. + +### 7.1 Admin Navigation Structure + +```mermaid +graph TD + subgraph "Admin Entry Point" + A["admin.js"] --> AI["Admin Vue Instance"] + AI --> AR["Admin Router"] + AI --> AP["Admin Layout"] + end + + subgraph "Admin Navigation" + AP --> AN["AdminNavbar"] + AP --> AM["AdminMenu"] + end + + subgraph "Admin Pages" + AR --> AD["AdminDashboard"] + AR --> AE["AdminEmailSending"] + AR --> AU["AdminUserList"] + AR --> AS["AdminSyncSetting"] + AR --> AC["AdminSystemConfig"] + end + + subgraph "Admin Routes" + RT1["/admin_panel/"] + RT2["/admin_panel/email_sending"] + RT3["/admin_panel/users"] + RT4["/admin_panel/sync"] + RT5["/admin_panel/system_config"] + end + + AD --- RT1 + AE --- RT2 + AU --- RT3 + AS --- RT4 + AC --- RT5 + + classDef primary fill:#f9f9f9,stroke:#333,stroke-width:1px; + class A,AP,AN primary; +``` + +Sources: [frontend/src/admin.js:1-71](), [frontend/src/main.js:146-152]() + +## 8. Component Reusability + +The CSGHub platform employs several strategies to enhance component reusability across the application. + +### 8.1 Shared Components + +The platform registers key components globally to ensure they're available throughout the application without requiring imports in each component: + +```javascript +// SvgIcon component for consistent icon usage +app.component('SvgIcon', SvgIcon) + +// CsgButton component for consistent button styling +app.component('CsgButton', CsgButton) + +// FlashMessage component for user notifications +app.component('FlashMessage', FlashMessage) +``` + +Sources: [frontend/src/main.js:160-162]() + +### 8.2 Element Plus Integration + +The navigation system extensively uses Element Plus components with custom styling to maintain consistency while benefiting from the functionality of a mature UI library: + +1. `el-menu` and `el-menu-item` for the main navigation +2. `el-dropdown` for user and language menus +3. `el-drawer` for the mobile navigation menu +4. `el-alert` for user notifications + +These components are customized using CSS variables and custom styles to match the application's design language. + +Sources: [frontend/src/components/navbar/Navbar.vue:19-39](), [frontend/src/components/navbar/Navbar.vue:44-57](), [frontend/src/components/navbar/Navbar.vue:258-285]() \ No newline at end of file diff --git a/docs/development/16_Internationalization_i18n.md b/docs/development/16_Internationalization_i18n.md new file mode 100644 index 000000000..c778df725 --- /dev/null +++ b/docs/development/16_Internationalization_i18n.md @@ -0,0 +1,292 @@ +# Internationalization (i18n) + +This document describes the internationalization (i18n) system used in CSGHub, which provides multi-language support throughout the application. The system enables the application to display text content in different languages based on user preferences. + +## Overview + +CSGHub implements internationalization using Vue I18n, a plugin that provides localization capabilities for Vue.js applications. The system currently supports English and Chinese languages, with a modular structure for organizing translation strings across different features of the application. + +```mermaid +flowchart TD + subgraph "i18n System" + I18n["Vue I18n Plugin"] --- LocaleFiles["Locale Files"] + I18n --- LangSwitch["Language Switching"] + + subgraph "Locale Files" + EN["English (en_js)"] --- ENAll["all.js"] + EN --- ENEndpoints["endpoints.js"] + EN --- ENOther["Other modules..."] + + ZH["Chinese (zh_js)"] --- ZHAll["all.js"] + ZH --- ZHEndpoints["endpoints.js"] + ZH --- ZHOther["Other modules..."] + end + end + + I18n --- Components["Vue Components"] + Components --- TransMethod["$t() Translation Method"] + +``` + +Sources: [frontend/src/locales/en_js/all.js](), [frontend/src/locales/zh_js/all.js](), [frontend/src/locales/en_js/endpoints.js](), [frontend/src/locales/zh_js/endpoints.js]() + +## Translation File Structure + +The translation files are organized in a modular way, with separate JavaScript files for different functional areas of the application. Each language has its own directory with equivalent translation files. + +```mermaid +graph TD + subgraph "locales Directory" + subgraph "en_js (English)" + ENAll["all.js (General translations)"] + ENEndpoints["endpoints.js (Endpoint-specific)"] + ENModels["models.js"] + ENDatasets["datasets.js"] + ENOther["Other modules..."] + end + + subgraph "zh_js (Chinese)" + ZHAll["all.js (General translations)"] + ZHEndpoints["endpoints.js (Endpoint-specific)"] + ZHModels["models.js"] + ZHDatasets["datasets.js"] + ZHOther["Other modules..."] + end + end + + MainJS["main.js"] --> I18nConfig["i18n Configuration"] + I18nConfig --> ENAll + I18nConfig --> ENEndpoints + I18nConfig --> ZHAll + I18nConfig --> ZHEndpoints +``` + +Sources: [frontend/src/locales/en_js/all.js](), [frontend/src/locales/zh_js/all.js](), [frontend/src/locales/en_js/endpoints.js](), [frontend/src/locales/zh_js/endpoints.js]() + +### Translation Files Content + +Each translation file exports JavaScript objects containing key-value pairs. The keys serve as identifiers for the text content, while the values contain the translated text in the respective language. + +For example, the English general translations file contains: + +```javascript +export const all = { + all: 'All', + defaultText: "```\nREADME file is empty, please download the file and add description content.\n```", + tip: "tip", + summary: "Summary", + files: "Files", + // ...many more translations +} +``` + +While the Chinese equivalent contains: + +```javascript +export const all = { + all: '全部应用', + defaultText: "```\nREADME文件内容为空，请下载文件，补充描述内容。\n```", + tip: "提示", + summary: "介绍", + files: "文件", + // ...many more translations +} +``` + +More specific modules like `endpoints.js` contain translations related to a specific feature: + +```javascript +export const endpoints = { + title: "Inference Endpoint", + resourceType: 'Resource Type', + replica: "Replica", + // ...more endpoint-specific translations +} +``` + +Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/zh_js/all.js:1-118](), [frontend/src/locales/en_js/endpoints.js:1-90](), [frontend/src/locales/zh_js/endpoints.js:1-90]() + +## Using Translations in Components + +Components can access translations using the `$t()` function provided by Vue I18n. The function takes a string key that references a specific translation. + +### Key-to-Component Flow + +```mermaid +sequenceDiagram + participant Component as "Vue Component" + participant I18n as "Vue I18n Plugin" + participant Locale as "Locale Files" + + Component->>I18n: $t('endpoints.title') + I18n->>Locale: Look up key in current locale + Locale-->>I18n: Return "Inference Endpoint" (EN) or "专属实例" (ZH) + I18n-->>Component: Display translated text +``` + +### Examples in Components + +Here's how translations are used in actual components: + +1. Basic text translation: + +```vue +

+ + {{ $t('endpoints.detail.endpointUrl') }} +

+``` + +2. Dynamic translations with parameters: + +```vue +

{{ $t('all.pleaseSelect', { value: 'username' }) }}

+``` + +3. Using translations in script blocks: + +```javascript +ElMessage.warning(error.value.msg || t('all.fetchError')) +``` + +Sources: [frontend/src/components/endpoints/EndpointPage.vue:6-7](), [frontend/src/components/datasets/ParquetViewer.vue:10-11](), [frontend/src/components/datasets/ParquetViewer.vue:44-45]() + +## Translation Key Organization + +Translation keys are organized hierarchically to maintain a logical structure. This helps with organization and reduces the chance of key conflicts. + +```mermaid +graph TD + subgraph "Translation Key Structure" + Root["Root Keys"] --> All["all.*"] + Root --> Endpoints["endpoints.*"] + Root --> Models["models.*"] + Root --> Datasets["datasets.*"] + + All --> AllGeneral["General UI text (all.confirm, all.cancel)"] + Endpoints --> EPDetail["endpoints.detail.*"] + Endpoints --> EPSettings["endpoints.settings.*"] + Endpoints --> EPPlayground["endpoints.playground.*"] + end + + subgraph "Component Usage" + Component1["EndpointPage.vue"] --> EPDetail + Component2["ParquetViewer.vue"] --> All + Component3["RepoSummary.vue"] --> AllGeneral + end +``` + +Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/en_js/endpoints.js:1-90](), [frontend/src/components/endpoints/EndpointPage.vue:1-142]() + +## Adding New Translations + +To add new translations to the system, developers need to: + +1. Identify the appropriate module file for the new translation +2. Add the new key-value pair to all supported language files +3. Use the new key in components with the `$t()` function + +### Translation Key Patterns + +The system uses consistent naming patterns for translation keys: + +| Pattern | Purpose | Example | +|---------|---------|---------| +| `module.noun` | General terms | `all.summary`, `all.files` | +| `module.verb` | Action labels | `all.add`, `all.update` | +| `module.section.item` | UI section items | `endpoints.detail.endpointUrl` | +| `module.section.action` | Section-specific actions | `endpoints.settings.changeVisibility` | + +Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/en_js/endpoints.js:1-90]() + +## Implementation in Components + +Here's a detailed look at how i18n is implemented in components: + +### Component Integration + +```mermaid +graph LR + subgraph "VueComponent" + Template["Template Section"] --> T["$t() calls"] + Script["Script Section"] --> UseI18n["useI18n() hook"] + UseI18n --> TFunction["t() function"] + end + + subgraph "I18nSystem" + I18nPlugin["Vue I18n Plugin"] + LocaleMessages["Locale Messages"] + end + + T --> I18nPlugin + TFunction --> I18nPlugin + I18nPlugin --> LocaleMessages +``` + +In Vue components, i18n can be used in both the template section via `$t()` and in the script section via the `useI18n()` hook: + +1. Template usage: +```vue +
+ {{ $t('all.downloadCount') }} +
+``` + +2. Script usage: +```javascript +import { useI18n } from 'vue-i18n' + +// In setup() +const { t } = useI18n() +ElMessage.warning(error.value.msg || t('all.fetchError')) +``` + +Sources: [frontend/src/components/shared/RepoSummary.vue:18-19](), [frontend/src/components/shared/RepoSummary.vue:121](), [frontend/src/components/datasets/ParquetViewer.vue:236]() + +### Dynamic Content with i18n + +Some components like `ParquetViewer.vue` use i18n with dynamic data, combining translated labels with application data: + +```vue +

{{ $t('all.subset') }}（{{ numSubsets }}）

+``` + +Sources: [frontend/src/components/datasets/ParquetViewer.vue:44]() + +### Loading Indicators with i18n + +Internationalized loading messages are used in components: + +```vue +
+ +
+``` + +Sources: [frontend/src/components/endpoints/EndpointPage.vue:71-73]() + +## Testing i18n Implementation + +The codebase includes tests for components that use i18n functionalities: + +```javascript +// Component test example +describe("EndpointSettings", () => { + it("mounts correctly", () => { + const wrapper = createWrapper(); + expect(wrapper.exists()).toBe(true); + }); +}); +``` + +When testing components that use i18n, the testing framework must include the i18n plugin setup to properly render translated content. + +Sources: [frontend/src/components/__tests__/endpoints/EndpointSettings.spec.js:109-114]() + +## Conclusion + +The internationalization system in CSGHub provides a robust framework for supporting multiple languages throughout the application. The modular approach to organizing translation files allows for easy maintenance and extension of the language support. The consistent use of the `$t()` function in components ensures that all user-facing text can be translated seamlessly. + +For information about the overall frontend architecture, see [Frontend Architecture](#5). + +For details about API integration that works alongside the i18n system, see [API Integration and Data Fetching](#5.3). \ No newline at end of file diff --git a/docs/development/17_API_Integration_and_Data_Fetching.md b/docs/development/17_API_Integration_and_Data_Fetching.md new file mode 100644 index 000000000..fba01fa0b --- /dev/null +++ b/docs/development/17_API_Integration_and_Data_Fetching.md @@ -0,0 +1,298 @@ +# API Integration and Data Fetching + +This page documents the API integration system and data fetching patterns used in the CSGHub frontend. It focuses on how frontend components communicate with the backend API, including the core utilities, authentication handling, and common data fetching patterns. + +## Core API Integration System + +The CSGHub frontend uses a centralized API client built on top of @vueuse/core's `createFetch`. This system provides a consistent interface for all API requests, handling authentication, error management, and localization automatically. + +```mermaid +flowchart TD + subgraph "Frontend Components" + NewEndpoint["NewEndpoint.vue"] + EndpointSettings["EndpointSettings.vue"] + FinetuneComponents["Finetune Components"] + EvaluationComponents["Evaluation Components"] + end + + subgraph "API Integration Layer" + useFetchApi["useFetchApi"] + refreshJWT["refreshJWT.js"] + auth["auth.js"] + end + + subgraph "Backend API" + GinRouter["Gin Router"] + Endpoints["/models/*/run"] + Finetunes["/models/*/finetune"] + Resources["/space_resources"] + RuntimeFrameworks["/runtime_framework"] + end + + NewEndpoint --> |"calls"| useFetchApi + EndpointSettings --> |"calls"| useFetchApi + FinetuneComponents --> |"calls"| useFetchApi + EvaluationComponents --> |"calls"| useFetchApi + + useFetchApi --> |"refreshes token"| refreshJWT + useFetchApi --> |"handles errors"| auth + + useFetchApi --> |"requests"| GinRouter + GinRouter --> |"routes to"| Endpoints + GinRouter --> |"routes to"| Finetunes + GinRouter --> |"routes to"| Resources + GinRouter --> |"routes to"| RuntimeFrameworks +``` + +Sources: +- [frontend/src/packs/useFetchApi.js:1-90]() +- [frontend/src/packs/auth.js:1-41]() + +### The useFetchApi Utility + +The `useFetchApi` utility centralizes all API communication, providing several key features: + +- **Automatic JWT token handling**: Refreshes tokens before requests +- **Internationalization support**: Passes user language preferences +- **Standardized error handling**: Consistent approach to error reporting +- **Session management**: Handles expired sessions with re-login prompts + +```mermaid +sequenceDiagram + participant Component as "Vue Component" + participant FetchAPI as "useFetchApi" + participant JWT as "refreshJWT" + participant Backend as "Backend API" + + Component->>FetchAPI: Call API endpoint + FetchAPI->>JWT: Check & refresh token if needed + JWT-->>FetchAPI: Updated token + FetchAPI->>Backend: Make request with Authorization header + Backend-->>FetchAPI: Response (success/error) + + alt Success response + FetchAPI-->>Component: Return data.value + else Error response (401) + FetchAPI->>FetchAPI: Show re-login dialog + else Other errors + FetchAPI-->>Component: Return error.value + end +``` + +Sources: +- [frontend/src/packs/useFetchApi.js:32-76]() + +## Common Data Fetching Patterns + +CSGHub implements several common patterns for data fetching that are reused across components. + +### Resource Fetching + +The `fetchResourcesInCategory` function retrieves cloud resources available for different deployment types, organizing them by payment mode and formatting them for display. + +```javascript +// Example of how components use the fetchResourcesInCategory utility +const fetchResources = async () => { + const categoryResources = await fetchResourcesInCategory(dataForm.value.cluster_id) + const firstAvailableResource = categoryResources.flatMap(item => item.options) + .find((item) => item.is_available) + // ... further processing of resources +} +``` + +Sources: +- [frontend/src/components/shared/deploy_instance/fetchResourceInCategory.js:1-53]() +- [frontend/src/components/endpoints/NewEndpoint.vue:397-408]() + +### Request-Response Structure + +All API calls follow a consistent pattern for handling responses and errors: + +```mermaid +flowchart TD + A["Component calls useFetchApi"] --> B["useFetchApi makes request"] + B --> C{"Response received"} + + C -->|"Success (2xx)"| D["Extract data.value"] + C -->|"Error (4xx/5xx)"| E["Extract error.value"] + C -->|"Auth Error (401)"| F["Show login dialog"] + + D --> G["Update component state"] + E --> H["Show error message"] + F --> I["Redirect to login"] + + G --> J["Update UI"] + H --> J +``` + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:551-566]() +- [frontend/src/packs/useFetchApi.js:64-77]() + +## Component API Integration Examples + +The following sections demonstrate how different components integrate with the API. + +### Creating Resources (Endpoints, Finetunes, Evaluations) + +Resource creation components follow a similar pattern: + +1. Collect user input in a form +2. Validate the form data +3. Submit the data to the appropriate API endpoint +4. Handle the response (success/failure) +5. Redirect on success or display error + +This pattern is evident in components like NewEndpoint, NewFinetune, and NewEvaluation. + +```mermaid +flowchart TD + A["User fills form"] --> B["Component validates data"] + B --> C{"Validation successful?"} + + C -->|"No"| D["Show validation errors"] + C -->|"Yes"| E["Call useFetchApi with form data"] + + E --> F{"API Response"} + F -->|"Success"| G["Show success message & redirect"] + F -->|"Error"| H["Show error message"] + + H --> A + D --> A +``` + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:511-567]() +- [frontend/src/components/finetune/NewFinetune.vue:333-367]() +- [frontend/src/components/evaluations/NewEvaluation.vue:453-488]() + +### Managing Resources (Settings/Updates) + +Components that manage existing resources typically: + +1. Fetch current resource data on mount +2. Allow users to modify settings +3. Submit updates through API calls +4. Handle success/failure states + +Below is an example from the EndpointSettings component: + +| API Operation | Endpoint | Method | Purpose | +|---------------|----------|--------|---------| +| Stop Endpoint | `/models/${modelId}/run/${endpointId}/stop` | PUT | Pause an endpoint | +| Restart Endpoint | `/models/${modelId}/run/${endpointId}/start` | PUT | Resume a stopped endpoint | +| Update Endpoint | `/models/${modelId}/run/${endpointId}` | PUT | Modify endpoint settings | +| Delete Endpoint | `/models/${modelId}/run/${endpointId}` | DELETE | Remove an endpoint | + +Sources: +- [frontend/src/components/endpoints/EndpointSettings.vue:397-563]() +- [frontend/src/components/finetune/FinetuneSettings.vue:284-379]() + +## The API Request Lifecycle + +The following diagram illustrates the complete lifecycle of an API request within the CSGHub frontend, from component initiation to UI updates: + +```mermaid +sequenceDiagram + participant Component as "Vue Component" + participant Store as "Pinia Store" + participant FetchAPI as "useFetchApi" + participant Auth as "Auth System" + participant Backend as "Backend API" + + Note over Component,Backend: Request Initialization + Component->>Component: User action or component mounted + Component->>FetchAPI: Call useFetchApi(endpoint, options) + + Note over FetchAPI,Auth: Request Preparation + FetchAPI->>Auth: beforeFetch hook checks authentication + Auth->>Auth: refreshJWT if needed + Auth-->>FetchAPI: Add Authorization header + FetchAPI->>FetchAPI: Add Accept-Language header + + Note over FetchAPI,Backend: Request Execution + FetchAPI->>Backend: HTTP request (GET/POST/PUT/DELETE) + Backend-->>FetchAPI: Response with data or error + + Note over FetchAPI,Component: Response Handling + alt Successful response (2xx) + FetchAPI-->>Component: Return {data, error: null} + Component->>Store: Update state if needed + Component->>Component: Update UI with data + else Error response (401) + FetchAPI->>FetchAPI: Call popupReloginDialog + FetchAPI->>Auth: logout() + else Other error response + FetchAPI-->>Component: Return {data: null, error} + Component->>Component: Display error message + end +``` + +Sources: +- [frontend/src/packs/useFetchApi.js:43-77]() +- [frontend/src/components/endpoints/NewEndpoint.vue:526-566]() + +## Common API Endpoints and Their Usage + +The CSGHub frontend interacts with several key API endpoints for different operations: + +| Category | Endpoint Pattern | Purpose | Example Component | +|----------|-----------------|---------|-------------------| +| Endpoints | `/models/{model_id}/run` | Create/manage inference endpoints | NewEndpoint, EndpointSettings | +| Finetunes | `/models/{model_id}/finetune` | Create/manage model fine-tuning | NewFinetune, FinetuneSettings | +| Evaluations | `/evaluations` | Create/manage model evaluations | NewEvaluation | +| Resources | `/space_resources` | Fetch available compute resources | Multiple components | +| Frameworks | `/models/{model_id}/runtime_framework` | Get compatible frameworks | Multiple components | +| Clusters | `/cluster` | List available clusters | Multiple components | +| Models | `/runtime_framework/models` | Search for models | Multiple components | + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:550-551]() +- [frontend/src/components/finetune/NewFinetune.vue:353-357]() +- [frontend/src/components/evaluations/NewEvaluation.vue:472-473]() +- [frontend/src/components/shared/deploy_instance/fetchResourceInCategory.js:19-20]() + +## Special Feature: Engine Arguments Management + +The system includes specialized handling for engine arguments used to configure model runtime behavior. This is implemented through the EngineArgs component and its supporting utility. + +```mermaid +flowchart TD + A["Parent Component"] -->|"pass engineArgs prop"| B["EngineArgs Component"] + B -->|"uses"| C["getInputTypeForEngineArg Utility"] + C -->|"determines"| D["Input Type (select/switch/text)"] + B -->|"emits changes"| E["update:changedArgs Event"] + A -->|"receives"| E + A -->|"includes in API request"| F["API Call with engine_args"] +``` + +Sources: +- [frontend/src/components/endpoints/EngineArgs.vue:1-127]() +- [frontend/src/packs/useEngineArgs.js:1-55]() +- [frontend/src/components/endpoints/NewEndpoint.vue:542-544]() + +This specialized system provides appropriate input controls based on parameter type and tracks changes to include only modified parameters in API requests. + +## Error Handling and Authentication + +The API integration system includes robust error handling, particularly for authentication issues: + +1. **401 Unauthorized Errors**: Trigger a re-login dialog +2. **Other Error Status Codes**: Return error information to the component +3. **Network Failures**: Captured and returned as errors + +For authentication, the system automatically: +- Refreshes JWT tokens before requests +- Handles token expiration +- Manages user logout + +Sources: +- [frontend/src/packs/useFetchApi.js:13-30]() +- [frontend/src/packs/useFetchApi.js:64-77]() +- [frontend/src/packs/auth.js:28-38]() + +## Summary + +The CSGHub API integration and data fetching system provides a consistent, robust approach to communicating with the backend. By centralizing core functionality in the `useFetchApi` utility, the system ensures consistent error handling, authentication, and response processing across all components. + +Components follow established patterns for resource creation, management, and deletion, with specialized utilities for common operations like resource fetching. This design promotes code reuse and maintainability while providing a consistent user experience for handling success and error states. \ No newline at end of file diff --git a/docs/development/18_Admin_Panel.md b/docs/development/18_Admin_Panel.md new file mode 100644 index 000000000..3b75884ac --- /dev/null +++ b/docs/development/18_Admin_Panel.md @@ -0,0 +1,313 @@ +# Admin Panel + +## Purpose and Scope + +The Admin Panel in CSGHub provides administrative capabilities for managing the platform, including system configuration, user management, model management, tag management, and broadcasting. This document covers the structure, architecture, and features of the Admin Panel, which is accessible only to users with admin privileges (those with the `admin` or `super_user` role). + +For information about other frontend components, see the [Frontend Architecture](#5) and [Component System and Navigation](#5.1) pages. + +## Access Control + +The Admin Panel is only accessible to authenticated users with admin privileges. The access mechanism is implemented in the main navigation component, where the admin panel link is conditionally displayed. + +```mermaid +flowchart TD + Start([User Request]) --> CheckLogin{"isLoggedIn?"} + CheckLogin -->|No| NoAccess["No Admin Panel Access"] + CheckLogin -->|Yes| CheckRoles{"isAdmin? (!actionLimited?)"} + CheckRoles -->|No| NoAccess + CheckRoles -->|Yes| ShowAdminLink["Display Admin Panel Link"] + ShowAdminLink --> AdminClick["User Clicks Admin Panel"] + AdminClick --> AdminPanel["Mount Admin Panel App"] +``` + +The admin privileges check is performed using computed properties in the UserStore that check for the presence of admin roles: + +Sources: +- [frontend/src/components/navbar/Navbar.vue:137-145]() +- [frontend/src/stores/UserStore.js:26-27]() + +## Architecture + +The Admin Panel is implemented as a dedicated Vue application within the CSGHub platform, with its own router, state management, and component hierarchy. This separation provides better maintainability and performance isolation. + +### Admin Panel Application Structure + +```mermaid +graph TD + AdminEntry["admin.js Entry Point"] --> CreateApp["Create Vue App with AdminLayout"] + CreateApp --> RegisterComponents["Register Icons & Components"] + RegisterComponents --> SetupPlugins["Setup Plugins (i18n, pinia, router)"] + SetupPlugins --> MountApp["Mount App to #admin"] + + subgraph "Core Components" + AdminLayout["AdminLayout.vue"] + AdminNavbar["AdminNavbar"] + AdminMenu["AdminMenu"] + FeatureComponents["Feature-specific Components"] + end + + subgraph "Routing System" + AdminRouter["ROUTERS Configuration"] + MenuSettings["MENU_SETTING Routes"] + ParentMenus["PARENT_NAME_LIST Grouping"] + end + + MountApp --> AdminLayout + AdminLayout --> AdminNavbar + AdminLayout --> AdminMenu + AdminLayout --> FeatureComponents + + FeatureComponents --> ApiCalls["useFetchApi() Calls"] + ApiCalls --> BackendAPI["Backend Admin API"] +``` + +Sources: +- [frontend/src/admin.js:46-70]() +- [frontend/src/components/admin_next/router/index.js:51-202]() + +## Navigation Structure + +The Admin Panel is organized into hierarchical menu groups that provide logical categorization of administrative features. + +### Admin Panel Menu Structure + +```mermaid +graph TD + AdminPanel["Admin Panel"] --> BaseConfig["Basic Settings"] + AdminPanel --> SystemMenu["System"] + AdminPanel --> AccountMenu["Account"] + AdminPanel --> HubMenu["Hub"] + + BaseConfig --> Dashboard["AdminDashboard"] + + SystemMenu --> SystemConfig["AdminSystemConfig"] + SystemMenu --> Broadcasts["AdminBroadcastList"] + + AccountMenu --> Users["AdminUserList"] + + HubMenu --> Sync["AdminSyncSetting"] + HubMenu --> Models["AdminModelList"] + HubMenu --> Serverless["AdminServerlessList"] + HubMenu --> Tags["AdminTagsList"] + HubMenu --> TagCategories["AdminTagCategoriesList"] + + Users --> UserDetail["AdminUserDetail (/:id)"] + Models --> ModelDetail["AdminModelDetail (/:namespace/:name)"] + Models --> ModelEdit["AdminModelEdit (/:namespace/:name/edit)"] + Serverless --> ServerlessForm["AdminServerlessForm (new/edit)"] + Serverless --> ServerlessDetail["AdminServerlessDetail (/:id)"] + Tags --> TagForm["AdminTagsForm (new/edit)"] + Tags --> TagDetail["AdminTagsDetail (/:id)"] + TagCategories --> TagCategoryForm["AdminTagCategoriesForm (new/edit)"] + Broadcasts --> BroadcastNew["AdminBroadcastNew"] + Broadcasts --> BroadcastEdit["AdminBroadcastEdit (/:id/edit)"] +``` + +Sources: +- [frontend/src/components/admin_next/router/index.js:51-202]() +- [frontend/src/components/admin_next/router/index.js:34-39]() + +## Features and Components + +The Admin Panel provides several administrative features, each implemented as a set of Vue components: + +### Dashboard + +The dashboard displays an overview of the platform and administrative information. It's the default landing page for the Admin Panel. + +Sources: +- [frontend/src/components/admin_next/router/index.js:51-60]() +- [frontend/src/main.js:64]() + +### User Management + +Allows administrators to view, edit, and manage user accounts. It includes components for listing users and viewing detailed user information. + +| Component | Purpose | +|-----------|---------| +| AdminUserList | Displays all users with search and filtering capabilities | +| AdminUserDetail | Shows detailed information for a single user | + +Sources: +- [frontend/src/components/admin_next/router/index.js:70-83]() +- [frontend/src/main.js:66-67]() + +### System Configuration + +Provides settings for configuring system-wide behavior. In addition to general system configuration, this section also includes broadcast message management. + +| Component | Purpose | +|-----------|---------| +| AdminSystemConfig | Contains system-wide settings | +| AdminBroadcastList | Manages broadcast messages shown to users | +| AdminBroadcastNew | Creates new broadcast messages | +| AdminBroadcastEdit | Edits existing broadcast messages | + +Sources: +- [frontend/src/components/admin_next/router/index.js:61-69]() +- [frontend/src/components/admin_next/router/index.js:184-201]() +- [frontend/src/main.js:68]() + +### Multi-Source Sync + +Provides settings for synchronizing with external sources including concurrent task settings and bandwidth limit management. + +Sources: +- [frontend/src/components/admin_next/router/index.js:84-91]() +- [frontend/src/main.js:69]() +- [frontend/src/locales/zh_js/admin.js:35-43]() +- [frontend/src/locales/en_js/admin.js:35-43]() + +### Model Management + +Allows administrators to view, edit, and manage models, including model weight settings and synchronization. + +| Component | Purpose | +|-----------|---------| +| AdminModelList | Lists all models with filtering and search capabilities | +| AdminModelDetail | Shows detailed information for a single model | +| AdminModelEdit | Provides form for editing model properties | + +Sources: +- [frontend/src/components/admin_next/router/index.js:92-111]() +- [frontend/src/locales/zh_js/admin.js:54-64]() +- [frontend/src/locales/en_js/admin.js:54-64]() + +### Serverless API Management + +Allows administrators to manage serverless API deployments, including deployment configuration and status management. + +| Component | Purpose | +|-----------|---------| +| AdminServerlessList | Lists serverless deployments | +| AdminServerlessForm | Creates/edits serverless deployments | +| AdminServerlessDetail | Shows details for a single deployment | + +Sources: +- [frontend/src/components/admin_next/router/index.js:112-137]() +- [frontend/src/components/admin_next/serverless/AdminServerlessList.vue:1-132]() +- [frontend/src/locales/zh_js/admin.js:65-90]() +- [frontend/src/locales/en_js/admin.js:65-90]() + +### Tag Management + +Allows administrators to manage tags and tag categories, which are used throughout the platform for content organization. + +| Component | Purpose | +|-----------|---------| +| AdminTagsList | Lists all tags with filtering capabilities | +| AdminTagsForm | Creates/edits tags | +| AdminTagsDetail | Shows details for a single tag | +| AdminTagCategoriesList | Lists tag categories | +| AdminTagCategoriesForm | Creates/edits tag categories | + +Sources: +- [frontend/src/components/admin_next/router/index.js:138-163]() +- [frontend/src/components/admin_next/router/index.js:164-183]() +- [frontend/src/locales/zh_js/admin.js:92-130]() +- [frontend/src/locales/en_js/admin.js:92-130]() + +## Internationalization Support + +The Admin Panel fully supports internationalization (i18n) with English and Chinese language options. Translation strings are defined in dedicated locale files. + +| Language | File Path | +|----------|-----------| +| English | frontend/src/locales/en_js/admin.js | +| Chinese | frontend/src/locales/zh_js/admin.js | + +The locale selection is determined by user preference stored in cookies, with a fallback to browser language detection. + +Sources: +- [frontend/src/admin.js:29-39]() +- [frontend/src/components/admin_next/router/index.js:24-31]() +- [frontend/src/locales/en_js/admin.js:1-143]() +- [frontend/src/locales/zh_js/admin.js:1-143]() + +## State Management + +The Admin Panel uses Pinia for state management, with UserStore providing user data and access control. A custom persist plugin ensures state is available across page refreshes. + +```mermaid +graph TD + UserStoreDefinition["useUserStore() Definition"] --> StateProps["State Properties (username, roles, etc.)"] + UserStoreDefinition --> ComputedProps["Computed Properties"] + + ComputedProps --> IsLoggedIn["isLoggedIn"] + ComputedProps --> IsAdmin["isAdmin"] + ComputedProps --> IsSuperUser["isSuperUser"] + ComputedProps --> CanChangeUsername["canChangeUsername"] + ComputedProps --> HasEmail["hasEmail"] + ComputedProps --> ActionLimited["actionLimited"] + + UserStoreDefinition --> Actions["Store Actions"] + Actions --> Initialize["initialize()"] + Actions --> ClearStore["clearStore()"] + + PiniaPlugin["customPersistPlugin"] --> LocalStorage["localStorage (persist)"] + UserStoreDefinition --> LocalStorage +``` + +Sources: +- [frontend/src/stores/UserStore.js:7-93]() +- [frontend/src/admin.js:38-39]() +- [frontend/src/packs/persistPinia.js:1-20]() + +## Implementation Details + +### Routing Configuration + +The Admin Panel routing is defined in `admin_next/router/index.js`. The routes are organized into menu groups (Basic Settings, System, Account, Hub) with parent-child relationships. + +```javascript +export const ROUTERS = [ + { + path: BASE_URL, + children: MENU_SETTING, + }, +]; +``` + +Sources: +- [frontend/src/components/admin_next/router/index.js:34-46]() +- [frontend/src/components/admin_next/router/index.js:215-220]() + +### Admin Panel Initialization + +The Admin Panel is initialized in `admin.js`, which creates and configures a Vue application with the AdminLayout component as its root: + +```javascript +const app = createApp(AdminLayout); +// Register components and plugins +app.mount("#admin"); +``` + +Sources: +- [frontend/src/admin.js:46-70]() + +### Component Reuse + +The Admin Panel uses shared components for consistent UI patterns: + +- Container component for layout structure +- Table component for data display +- Form components for data input +- SvgIcon component for icons +- CsgButton component for actions + +Sources: +- [frontend/src/admin.js:51-52]() +- [frontend/src/components/admin_next/serverless/AdminServerlessList.vue:86]() + +## Admin Panel vs Main Application + +The Admin Panel is built as a separate Vue application (`admin.js`) rather than as a component within the main application (`main.js`). This separation provides better isolation and maintainability, while allowing the Admin Panel to use the same UI components and styling as the main application. + +Sources: +- [frontend/src/admin.js:46-70]() +- [frontend/src/main.js:147-152]() + +## Conclusion + +The CSGHub Admin Panel provides a comprehensive set of administrative tools organized into a logical hierarchy. Its implementation as a separate Vue application with dedicated routing and state management ensures good separation of concerns while maintaining a consistent user experience with the rest of the platform. \ No newline at end of file diff --git a/docs/development/19_Backend_Architecture.md b/docs/development/19_Backend_Architecture.md new file mode 100644 index 000000000..08f5cf87c --- /dev/null +++ b/docs/development/19_Backend_Architecture.md @@ -0,0 +1,451 @@ +# Backend Architecture + +This document provides a technical overview of the CSGHub backend architecture, focusing on the server-side components that power the platform. This includes the web server, routing system, request handling, middleware, and the integration patterns with external services. For information about the frontend architecture, see [Frontend Architecture](#5). + +## 1. Overview + +The CSGHub backend is built using Go with the Gin web framework, providing high-performance HTTP request handling with a middleware-based architecture. The backend serves both API endpoints and server-rendered web pages, acting as the central coordination point between the user interface, database, and external services like S3 storage and the Starhub Server. + +```mermaid +graph TD + subgraph "CSGHub Backend" + GinEngine["Gin Engine"] + Router["Router"] + Middleware["Middleware Chain"] + Handlers["Handlers"] + + GinEngine --> Router + Router --> Middleware + Middleware --> Handlers + end + + subgraph "External Services" + StarhubServer["Starhub Server"] + DB["Database"] + S3["S3 Storage"] + end + + subgraph "Clients" + WebUI["Web UI"] + GitCLI["Git CLI"] + SDK["CSGHub SDK"] + end + + WebUI --> GinEngine + GitCLI --> GinEngine + SDK --> GinEngine + + Handlers --> StarhubServer + Handlers --> DB + Handlers --> S3 +``` + +Sources: [internal/routes/router.go:33-95]() + +## 2. Router System + +The router system uses Gin's group-based routing to organize endpoints by resource type and authentication requirements. The main router initialization happens in the `Initialize` function, which sets up the server, configures logging, and registers routes. + +### 2.1 Router Initialization + +The router initialization process follows these steps: + +1. Create a new Gin engine +2. Configure trusted proxies and logging +3. Register global middleware +4. Initialize handler registries +5. Set up static file serving +6. Register API and view routes +7. Configure 404 handling + +```mermaid +sequenceDiagram + participant App as "Application" + participant Router as "Router" + participant Handlers as "Handlers Registry" + participant Middleware as "Middleware" + + App->>Router: Initialize(svcCtx) + Router->>Router: Create Gin Engine + Router->>Router: Configure Logging + Router->>Middleware: Register Global Middleware + Router->>Handlers: Initialize Handler Registries + Router->>Router: setupStaticRouter() + Router->>Router: setupApiRouter() + Router->>Router: setupViewsRouter() + Router->>Router: setupNotFoundRouter() + Router-->>App: Return Configured Gin Engine +``` + +Sources: [internal/routes/router.go:33-95]() + +### 2.2 Route Organization + +Routes are organized by resource type (models, datasets, codes, spaces, etc.) and are registered in separate functions. Each resource type has its own route group, and routes are further divided into public and authenticated routes. + +```mermaid +graph TD + Router["Router"] + + Router --> ApiRoutes["API Routes (/internal_api)"] + Router --> ModelRoutes["Model Routes (/models)"] + Router --> DatasetRoutes["Dataset Routes (/datasets)"] + Router --> CodeRoutes["Code Routes (/codes)"] + Router --> SpaceRoutes["Space Routes (/spaces)"] + Router --> EndpointRoutes["Endpoint Routes"] + Router --> FinetuneRoutes["Finetune Routes"] + Router --> OrgRoutes["Organization Routes"] + + subgraph "Common Route Pattern" + PublicRoutes["Public Routes"] + AuthRoutes["Authenticated Routes"] + + AuthMiddleware["Auth Middleware"] + + PublicRoutes --> ResourceHandler["Resource Handler"] + AuthRoutes --> AuthMiddleware + AuthMiddleware --> ResourceHandler + end +``` + +Sources: [internal/routes/router.go:205-230](), [internal/routes/models.go](), [internal/routes/datasets.go](), [internal/routes/codes.go](), [internal/routes/spaces.go]() + +## 3. Middleware System + +The Gin framework's middleware pattern is used extensively in CSGHub to handle cross-cutting concerns like authentication, logging, and error handling. Middleware is registered globally for the entire application and can also be applied to specific route groups. + +### 3.1 Global Middleware + +The following middleware is applied globally to all requests: + +1. Error recovery middleware (with custom logging) +2. Authentication middleware +3. Cache control middleware +4. Request logging middleware + +```mermaid +graph LR + Request["HTTP Request"] + Response["HTTP Response"] + + Request --> Recovery["Error Recovery Middleware"] + Recovery --> Auth["Authentication Middleware"] + Auth --> Cache["Cache Control Middleware"] + Cache --> Logger["Request Logging Middleware"] + Logger --> Handler["Route Handler"] + Handler --> Response +``` + +Sources: [internal/routes/router.go:61-72]() + +### 3.2 Route-Specific Middleware + +For protected routes, additional middleware is applied to check if a user is authenticated before allowing access to specific resources. + +```mermaid +flowchart LR + Request["HTTP Request to Protected Route"] + CheckUser["CheckCurrentUser Middleware"] + Handler["Route Handler"] + Unauthorized["401 Unauthorized"] + + Request --> CheckUser + CheckUser -- "User Authenticated" --> Handler + CheckUser -- "User Not Authenticated" --> Unauthorized +``` + +Sources: [internal/routes/models.go:22-23](), [internal/routes/datasets.go:16-17](), [internal/routes/codes.go:22-23](), [internal/routes/spaces.go:16-17]() + +## 4. Handler Architecture + +The handler architecture follows a hierarchical pattern with handler registries that manage related handlers. Handlers are organized into interfaces and implementations, with common functionality extracted into base handlers. + +### 4.1 Handler Registries + +Handler registries serve as collections of related handlers and are initialized during application startup. There are two main types of handler registries: + +1. `FrontendHandlerRegistry` - Handles API requests +2. `RenderHandlerRegistry` - Handles view rendering + +```mermaid +classDiagram + class HandlersRegistry { + +FrontendHandlers: FrontendHandlerRegistry + +RenderHandler: RenderHandlerRegistry + +Config: Config + } + + class RenderHandlerRegistry { + +ErrorHandler: ErrorHandler + +ModelHandler: ModelHandler + +DatasetHandler: DatasetHandler + +CodeHandler: CodeHandler + +SpaceHandler: SpaceHandler + +EndpointHandler: EndpointHandler + +FinetuneHandler: FinetuneHandler + +SessionHandler: SessionHandler + +OrganizationHandler: OrganizationHandler + +CollectionsHandler: CollectionsHandler + +ProfileHandler: ProfileHandler + +SettingHandler: SettingHandler + +AdminHandler: AdminHandler + +ResourceConsoleHandler: ResourceConsoleHandler + +PromptsHandler: PromptsHandler + +EvaluationHandler: EvaluationHandler + } + + HandlersRegistry *-- RenderHandlerRegistry +``` + +Sources: [internal/routes/router.go:26-31](), [internal/handlers/render/registry.go:9-26]() + +### 4.2 Base Handler Pattern + +Resource handlers follow a base handler pattern, where common functionality is implemented in a base handler and specialized behavior is added in resource-specific handlers. + +```mermaid +classDiagram + class BaseHandler { + <> + +List(ctx) + +Detail(ctx) + +Files(ctx) + +Blob(ctx) + +Commits(ctx) + +Commit(ctx) + +NewFile(ctx) + +UploadFile(ctx) + +EditFile(ctx) + +Settings(ctx) + +Billing(ctx) + +Logs(ctx) + +Community(ctx) + +New(ctx) + } + + class BaseHandlerImpl { + -resourceType: string + +List(ctx) + +Detail(ctx) + +Files(ctx) + +Blob(ctx) + +Commits(ctx) + +Commit(ctx) + +NewFile(ctx) + +UploadFile(ctx) + +EditFile(ctx) + +Settings(ctx) + +Billing(ctx) + +Logs(ctx) + +Community(ctx) + +New(ctx) + -renderShow(ctx, actionName, defaultTab, extraData) + -addResourceSpecificData(ctx, data) + } + + class ModelHandler { + <> + } + + class DatasetHandler { + <> + } + + class CodeHandler { + <> + } + + class SpaceHandler { + <> + } + + BaseHandler <|.. BaseHandlerImpl + BaseHandler <|-- ModelHandler + BaseHandler <|-- DatasetHandler + BaseHandler <|-- CodeHandler + BaseHandler <|-- SpaceHandler + + ModelHandler <|.. ModelHandlerImpl + DatasetHandler <|.. DatasetHandlerImpl + CodeHandler <|.. CodeHandlerImpl + SpaceHandler <|.. SpaceHandlerImpl + + BaseHandlerImpl <|-- ModelHandlerImpl + BaseHandlerImpl <|-- DatasetHandlerImpl + BaseHandlerImpl <|-- CodeHandlerImpl + BaseHandlerImpl <|-- SpaceHandlerImpl +``` + +Sources: [internal/handlers/render/repo.go:39-163](), [internal/handlers/render/models.go](), [internal/handlers/render/datasets.go](), [internal/handlers/render/codes.go](), [internal/handlers/render/spaces.go]() + +## 5. Authentication and Session Management + +CSGHub uses JWT (JSON Web Tokens) for authentication and session management. The JWT contains user information and is used to authenticate requests to protected resources. + +### 5.1 JWT Token Flow + +The JWT token system works as follows: + +1. User logs in and receives a JWT token +2. Token is stored in a cookie and used for subsequent requests +3. Token can be refreshed to extend the session +4. Token contains user information, including permissions and organizations + +```mermaid +sequenceDiagram + participant User + participant CSGHub as "CSGHub Backend" + participant Server as "Starhub Server" + + User->>CSGHub: Login Request + CSGHub->>Server: Create JWT Token Request + Server-->>CSGHub: JWT Token Response + CSGHub->>CSGHub: Set Cookie with Token + CSGHub-->>User: Login Response with Token + + User->>CSGHub: Protected Resource Request + CSGHub->>CSGHub: Verify Token in Middleware + CSGHub-->>User: Protected Resource Response + + User->>CSGHub: Refresh Token Request + CSGHub->>Server: Create New JWT Token + Server-->>CSGHub: New JWT Token + CSGHub->>CSGHub: Update Cookie + CSGHub-->>User: New Token Response +``` + +Sources: [internal/handlers/frontend/token.go:34-59](), [pkg/server/types/jwt.go]() + +### 5.2 Authentication Middleware + +The authentication middleware is responsible for verifying JWT tokens and extracting user information for protected routes. + +Table: Authentication Middleware Components + +| Component | Description | +|-----------|-------------| +| `AuthMiddleware` | Main middleware that validates JWT tokens | +| `CheckCurrentUser` | Ensures a user is authenticated for protected routes | +| `JwtUtils` | Utility for working with JWT tokens | +| `GetCurrentUser` | Extracts user information from JWT token | + +Sources: [internal/routes/router.go:69](), [internal/routes/models.go:23](), [internal/handlers/frontend/token.go:35]() + +## 6. View Rendering + +CSGHub uses server-side rendering with Go templates for HTML views. The template system is configured to use layouts and partials for consistent page structure. + +### 6.1 Template System + +The template system uses the `multitemplate.Renderer` to manage templates with layouts and partials: + +```mermaid +graph TD + subgraph "Template System" + TemplateRenderer["multitemplate.Renderer"] + Layouts["Layouts (base, navbar, footer)"] + Pages["Page Templates"] + + TemplateRenderer --> Layouts + TemplateRenderer --> Pages + + Handler["Render Handler"] --> TemplateRenderer + end + + subgraph "Route Handling" + Request["HTTP Request"] + Router["Router"] + RenderFunction["RenderTemplate Function"] + + Request --> Router + Router --> Handler + Handler --> RenderFunction + RenderFunction --> TemplateRenderer + end +``` + +Sources: [internal/routes/router.go:106-203]() + +## 7. Static File Serving + +The backend serves static files from the embedded frontend distribution. Files are served from memory without requiring external file access. + +```mermaid +graph LR + subgraph "Static File Serving" + Request["Static File Request"] + FileServer["http.FileServer"] + EmbeddedFS["Embedded FS (frontend.Dist)"] + + Request --> StaticMiddleware["Static File Middleware"] + StaticMiddleware --> FileServer + FileServer --> EmbeddedFS + EmbeddedFS --> Response["File Response"] + end +``` + +Sources: [internal/routes/router.go:232-250]() + +## 8. API Integration + +The backend provides internal API endpoints for use by the frontend, using a consistent pattern for API responses. + +### 8.1 Internal API Endpoints + +The internal API endpoints are registered in the `setupApiRouter` function and include: + +- `/internal_api/ping` - Health check endpoint +- `/internal_api/users/jwt_token` - Token refresh endpoint +- `/internal_api/upload` - File upload endpoint + +```mermaid +graph TD + subgraph "Internal API" + ApiGroup["/internal_api"] + Ping["/ping (PingHandler.Ping)"] + JwtToken["/users/jwt_token (TokenHandler.RefreshToken)"] + Upload["/upload (UploadHandler.Create)"] + + ApiGroup --> Ping + ApiGroup --> JwtToken + ApiGroup --> Upload + end + + subgraph "Other Endpoints" + Resolve["/:repo_type/:namespace/:name/resolve/:branch/*path"] + Locale["/zh|en/settings/locale"] + end +``` + +Sources: [internal/routes/router.go:252-265]() + +## 9. Error Handling + +The backend includes comprehensive error handling with custom recovery middleware and structured logging. + +```mermaid +flowchart TD + Request["HTTP Request"] + + subgraph "Error Handling System" + RecoveryMiddleware["Custom Recovery Middleware"] + ErrorLogging["Structured Error Logging (slog)"] + ErrorResponse["Error Response Generation"] + end + + Request --> RecoveryMiddleware + RecoveryMiddleware -- "No Error" --> Handler["Route Handler"] + RecoveryMiddleware -- "Error Occurs" --> ErrorLogging + ErrorLogging --> ErrorResponse + ErrorResponse --> Response["HTTP 500 Response"] + + Handler -- "Handler Error" --> ErrorLogging +``` + +Sources: [internal/routes/router.go:61-68]() + +## Conclusion + +The CSGHub backend architecture follows a well-structured design pattern based on the Gin web framework. It uses middleware extensively for cross-cutting concerns, implements a hierarchical handler structure for code reuse, and provides both API endpoints and server-rendered views. The authentication system is based on JWT tokens, and the template system uses layouts and partials for consistent page rendering. + +This architecture enables the backend to efficiently handle a variety of request types while maintaining separation of concerns between authentication, routing, and business logic. \ No newline at end of file diff --git a/docs/development/1_CSGHub_Overview.md b/docs/development/1_CSGHub_Overview.md new file mode 100644 index 000000000..47a33e12f --- /dev/null +++ b/docs/development/1_CSGHub_Overview.md @@ -0,0 +1,457 @@ +# CSGHub Overview + +CSGHub is an open-source platform designed for managing Large Language Model (LLM) assets. It provides a comprehensive solution for handling the entire lifecycle of LLMs and related assets such as datasets, spaces, and code repositories. CSGHub enables users to upload, download, store, verify, and distribute LLM assets through multiple interfaces: a web UI, Git command line, natural language chatbot, or the CSGHub SDK. + +This document provides a high-level overview of the CSGHub architecture and its core components. For more detailed information about specific subsystems, please refer to the dedicated pages for [Repository Management System](#2), [Model Deployment and Fine-tuning](#3), [User and Authentication System](#4), and [Frontend Architecture](#5). + +## Platform Architecture + +CSGHub implements a modern web application architecture with clear separation between frontend and backend components, connected through standardized APIs. The platform features microservice submodules that can be deployed either together or separately, making it suitable for both cloud and on-premise deployments. + +```mermaid +flowchart TB + subgraph "User Access Methods" + WebUI["Web Interface"] + GitCLI["Git CLI"] + SDK["CSGHub SDK"] + ChatBot["Natural Language Chatbot"] + end + + subgraph "CSGHub Platform" + Frontend["Frontend Application
(Vue.js)"] + + subgraph "Backend Services" + APIServer["API Server
(Go/Gin)"] + UserServer["User & Auth Service"] + StarhubServer["Starhub Server
(Resource Management)"] + end + + subgraph "Storage" + Database[(PostgreSQL/SQLite)] + S3["S3-Compatible Storage"] + GitStorage["Git Storage"] + end + end + + WebUI --> Frontend + GitCLI --> APIServer + SDK --> APIServer + ChatBot --> APIServer + + Frontend --> APIServer + APIServer --> UserServer + APIServer --> StarhubServer + APIServer --> Database + APIServer --> S3 + StarhubServer --> GitStorage + StarhubServer --> S3 +``` + +Sources: [README.md:12-14](frontend/src/components/navbar/Navbar.vue), [go.mod:1-19](go.mod), [config/config.go:14-40](config/config.go) + +## Core Components + +CSGHub is built with a modular architecture consisting of several key components: + +### 1. Frontend Application + +The frontend is a Vue.js-based single-page application that provides the web interface for users to interact with the platform. It includes: + +- User interface components (built with Element Plus) +- Internationalization support (English and Chinese) +- State management using Pinia +- Client-side routing with Vue Router + +### 2. Backend API Server + +The backend is primarily written in Go using the Gin web framework and provides APIs for: + +- Repository management (models, datasets, code, spaces) +- User and organization management +- Authentication and authorization +- Resource deployment (endpoints, fine-tuning) + +### 3. Storage Systems + +CSGHub uses multiple storage solutions: + +- Relational database (PostgreSQL or SQLite) for metadata and user information +- S3-compatible storage for binary assets and large files +- Git-based storage for version control of repositories + +### 4. Starhub Server + +A dedicated service for managing model deployment, inference endpoints, and fine-tuning instances. + +```mermaid +flowchart TB + subgraph "Frontend Components" + UI["Web UI"] + Router["Vue Router"] + State["Pinia State Store"] + i18n["Internationalization (i18n)"] + end + + subgraph "Backend Components" + API["Gin API Router"] + AuthMiddleware["Authentication Middleware"] + Handlers["Request Handlers"] + Database["Database Access Layer"] + S3Client["S3 Storage Client"] + end + + subgraph "Resource Management" + StarhubServer["Starhub Server"] + EndpointMgmt["Endpoint Management"] + FinetuneMgmt["Finetune Management"] + end + + UI --> Router + UI --> State + UI --> i18n + + Router --> API + API --> AuthMiddleware + AuthMiddleware --> Handlers + Handlers --> Database + Handlers --> S3Client + Handlers --> StarhubServer + + StarhubServer --> EndpointMgmt + StarhubServer --> FinetuneMgmt +``` + +Sources: [frontend/src/main.js:1-167](frontend/src/main.js), [config/config.go:14-40](config/config.go), [README.md:12-18](README.md) + +## Repository System + +The repository system is the core of CSGHub, providing management of various LLM assets: + +### Repository Types + +| Repository Type | Description | Primary Assets | +|----------------|-------------|----------------| +| Models | Large language models and their artifacts | Model weights, tokenizers, configuration files | +| Datasets | Training and evaluation data | Structured datasets, raw data, metadata | +| Code | Source code for LLM applications | Python scripts, notebooks, training code | +| Spaces | Interactive applications | Gradio and Streamlit applications | +| Collections | Curated groups of repositories | References to other repositories | + +### Repository Management + +Repositories in CSGHub can be accessed and managed through multiple methods: + +1. **Web Interface**: Browse, create, view, and manage repositories through the web UI +2. **Git CLI**: Use standard Git commands for version control and file management +3. **CSGHub SDK**: Programmatic access through the Python SDK +4. **API**: Direct API access for custom integrations + +```mermaid +flowchart TB + User([User]) + + subgraph "Access Methods" + WebUI["Web UI"] + Git["Git CLI"] + API["REST API"] + PythonSDK["Python SDK"] + end + + User --> WebUI + User --> Git + User --> PythonSDK + PythonSDK --> API + + subgraph "Repository System" + RepoHandler["Repository Handler"] + UserAuth["User Authentication"] + + subgraph "Repository Types" + Models["Models"] + Datasets["Datasets"] + Code["Code"] + Spaces["Spaces"] + Collections["Collections"] + end + + VersionControl["Git Version Control"] + S3Storage["Large File Storage (S3)"] + end + + WebUI --> RepoHandler + Git --> RepoHandler + API --> RepoHandler + + RepoHandler --> UserAuth + RepoHandler --> Models + RepoHandler --> Datasets + RepoHandler --> Code + RepoHandler --> Spaces + RepoHandler --> Collections + + Models --> VersionControl + Models --> S3Storage + Datasets --> VersionControl + Datasets --> S3Storage + Code --> VersionControl + Spaces --> VersionControl +``` + +Sources: [frontend/src/main.js:32-51](frontend/src/main.js), [README.md:12-14](README.md), [frontend/src/components/navbar/Navbar.vue:159-195](frontend/src/components/navbar/Navbar.vue) + +## User and Authentication System + +CSGHub implements a comprehensive user management and authentication system with the following features: + +1. **User accounts**: Registration, login, profile management +2. **Organizations**: Team-based access and repository ownership +3. **Role-based access control**: Different permission levels (admin, user) +4. **Authentication methods**: Username/password, OAuth, and API tokens +5. **SSH key management**: For secure Git operations + +```mermaid +flowchart TB + subgraph "User Interface" + Login["Login/Register Page"] + Profile["User Profile"] + Settings["User Settings"] + OrgSettings["Organization Settings"] + end + + subgraph "Authentication Flow" + Auth["Authentication System"] + JWT["JWT Token Handler"] + Session["Session Management"] + end + + subgraph "User Store" + UserStore["UserStore (Pinia)"] + Persist["Persistent Storage"] + end + + Login --> Auth + Auth --> JWT + JWT --> Session + + Profile --> UserStore + Settings --> UserStore + OrgSettings --> UserStore + + UserStore --> Persist + UserStore --> Auth + + Session --> UserStore +``` + +Sources: [frontend/src/stores/UserStore.js:1-95](frontend/src/stores/UserStore.js), [frontend/src/components/navbar/Navbar.vue:60-222](frontend/src/components/navbar/Navbar.vue), [frontend/src/packs/persistPinia.js:1-20](frontend/src/packs/persistPinia.js) + +## Resource Management + +CSGHub provides specialized tools for AI model deployment, fine-tuning, and evaluation: + +### Model Inference (Endpoints) + +Users can deploy models as endpoints for inference, with options for: +- Dedicated or shared instances +- Resource configuration +- Framework selection +- Region selection +- Interactive testing via playground interface + +### Model Fine-tuning + +CSGHub supports model fine-tuning with: +- Multiple fine-tuning frameworks (like LLaMA Factory) +- Dataset selection and preview +- Parameter configuration +- Resource allocation +- Performance monitoring + +### Model Evaluation + +The platform includes tools for comparing and evaluating models using standard benchmarks and custom datasets. + +```mermaid +flowchart TB + subgraph "Resource Console" + ResourceConsole["Resource Management Console"] + EndpointsList["Endpoints List"] + FinetuneList["Finetune Instances List"] + EvaluationList["Evaluation List"] + end + + subgraph "Endpoint System" + EndpointCreate["Create Endpoint"] + EndpointConfig["Configure Resources"] + EndpointDeploy["Deploy Model"] + EndpointMonitor["Monitor Performance"] + EndpointPlayground["Test in Playground"] + end + + subgraph "Finetune System" + FinetuneCreate["Create Finetune Instance"] + FinetuneConfig["Configure Parameters"] + FinetuneDataset["Select Dataset"] + FinetuneTrain["Train Model"] + FinetuneMonitor["Monitor Training"] + end + + subgraph "Evaluation System" + EvalCreate["Create Evaluation"] + EvalBenchmark["Select Benchmark"] + EvalRun["Run Evaluation"] + EvalResults["View Results"] + end + + ResourceConsole --> EndpointsList + ResourceConsole --> FinetuneList + ResourceConsole --> EvaluationList + + EndpointsList --> EndpointCreate + EndpointCreate --> EndpointConfig + EndpointConfig --> EndpointDeploy + EndpointDeploy --> EndpointMonitor + EndpointDeploy --> EndpointPlayground + + FinetuneList --> FinetuneCreate + FinetuneCreate --> FinetuneConfig + FinetuneCreate --> FinetuneDataset + FinetuneConfig --> FinetuneTrain + FinetuneDataset --> FinetuneTrain + FinetuneTrain --> FinetuneMonitor + + EvaluationList --> EvalCreate + EvalCreate --> EvalBenchmark + EvalBenchmark --> EvalRun + EvalRun --> EvalResults +``` + +Sources: [frontend/src/main.js:45-58](frontend/src/main.js), [docs/csghub_saas_en.md:90-178](docs/csghub_saas_en.md), [docs/csghub_saas_zh.md:90-178](docs/csghub_saas_zh.md) + +## Internationalization + +CSGHub provides comprehensive internationalization support for multiple languages: + +1. **Default languages**: English and Chinese +2. **Language detection**: Automatically detects browser language preference +3. **Language switching**: Users can manually switch between available languages +4. **Localized components**: All UI components and messages are localized + +```mermaid +flowchart TB + subgraph "i18n System" + I18n["Vue i18n"] + EnLocale["English Locale"] + ZhLocale["Chinese Locale"] + Cookies["Language Cookie"] + end + + subgraph "UI Components" + Navbar["Navigation Bar"] + RepoUI["Repository UI"] + Settings["Settings Page"] + Messages["System Messages"] + end + + Browser["Browser Language"] --> I18n + LanguageSelector["Language Selector"] --> I18n + + I18n --> EnLocale + I18n --> ZhLocale + I18n --> Cookies + + I18n --> Navbar + I18n --> RepoUI + I18n --> Settings + I18n --> Messages +``` + +Sources: [frontend/src/main.js:132-139](frontend/src/main.js), [frontend/src/components/navbar/Navbar.vue:44-57](frontend/src/components/navbar/Navbar.vue) + +## Deployment Options + +CSGHub can be deployed in several ways: + +| Deployment Method | Description | Use Case | +|------------------|-------------|----------| +| Docker All-in-One | Single container with all components | Quick experimentation, small-scale deployments | +| Docker Compose | Multi-container deployment | Development environments, medium deployments | +| Helm | Kubernetes-based deployment | Production, scalable environments | +| SaaS | Cloud service at opencsg.com | Immediate access without infrastructure | + +Sources: [README.md:39-47](README.md), [docs/release_notes.md:12-15](docs/release_notes.md) + +## Version History + +CSGHub follows a regular release cycle with version updates. Key milestones include: + +- v1.3.0 (2025.01.15): Added tag management, improved filters, SGLang support +- v1.2.0 (2024.12.15): Added model evaluation, redesigned admin UI +- v1.1.0 (2024.11.15): Enhanced multi-sync features, admin improvements +- v1.0.0 (2024.10.15): Stable multi-sync function, Docker for ARM, region support +- v0.9.0 (2024.09.15): Migrated to Go, added Collections feature + +For a complete history, see the [release notes](docs/release_notes.md). + +Sources: [docs/release_notes.md:3-31](docs/release_notes.md) + +## Technical Implementation + +CSGHub is built using modern web technologies: + +### Frontend +- Vue.js 3 as the frontend framework +- Pinia for state management +- Vue Router for client-side routing +- Element Plus for UI components +- Internationalization through Vue i18n + +### Backend +- Go programming language +- Gin web framework for routing and API +- PostgreSQL or SQLite for database storage +- S3-compatible storage for binary assets +- JWT for authentication + +```mermaid +flowchart TB + subgraph "Frontend Technologies" + Vue["Vue.js 3"] + Pinia["Pinia State Management"] + VueRouter["Vue Router"] + ElementPlus["Element Plus UI"] + I18n["Vue i18n"] + end + + subgraph "Backend Technologies" + Go["Go Language"] + Gin["Gin Web Framework"] + BunORM["Bun ORM"] + Cobra["Cobra CLI"] + end + + subgraph "External Services" + DB["PostgreSQL/SQLite"] + S3["S3 Storage"] + Git["Git Storage"] + end + + Vue --> Pinia + Vue --> VueRouter + Vue --> ElementPlus + Vue --> I18n + + Go --> Gin + Go --> BunORM + Go --> Cobra + + BunORM --> DB + Go --> S3 + Go --> Git +``` + +Sources: [frontend/package.json:14-43](frontend/package.json), [go.mod:5-18](go.mod), [frontend/src/main.js:1-8](frontend/src/main.js) + +## Conclusion + +CSGHub provides a comprehensive platform for managing LLM assets throughout their lifecycle. Its modular architecture and multiple deployment options make it suitable for a wide range of use cases, from individual research to enterprise-scale model management. The platform continues to evolve with regular updates and new features to support the growing needs of the LLM community. \ No newline at end of file diff --git a/docs/development/20_Routing_and_Middleware.md b/docs/development/20_Routing_and_Middleware.md new file mode 100644 index 000000000..a56660ef1 --- /dev/null +++ b/docs/development/20_Routing_and_Middleware.md @@ -0,0 +1,317 @@ +# Routing and Middleware + +This page documents the routing system and middleware chain in CSGHub's backend architecture. It covers how HTTP requests are handled, routed to the appropriate handlers, and processed through middleware components. For information about repository-specific handlers, see [Repository Handlers](#6.2). + +## Overview of the Routing System + +CSGHub uses the Gin framework for HTTP routing and middleware management. The routing system is responsible for directing incoming requests to the appropriate handlers based on URL patterns, HTTP methods, and other request characteristics. + +```mermaid +flowchart TB + subgraph "Request Processing Flow" + req["HTTP Request"] --> router["Gin Router"] + router --> middleware["Middleware Chain"] + middleware --> handlers["Route Handlers"] + handlers --> resp["HTTP Response"] + end + + subgraph "Middleware Components" + middl1["Recovery Middleware"] + middl2["Auth Middleware"] + middl3["Cache Control Middleware"] + middl4["Logging Middleware"] + middl5["Config Injection Middleware"] + end + + subgraph "Router Groups" + rg1["Static Routes"] + rg2["API Routes (/internal_api)"] + rg3["View Routes"] + rg4["Resolve Routes"] + rg5["Locale Routes"] + rg6["404 Handler"] + end + + middleware --> middl1 + middleware --> middl2 + middleware --> middl3 + middleware --> middl4 + middleware --> middl5 + + handlers --> rg1 + handlers --> rg2 + handlers --> rg3 + handlers --> rg4 + handlers --> rg5 + handlers --> rg6 +``` + +Sources: [internal/routes/router.go:33-96]() + +## Router Initialization + +The router initialization process sets up the Gin engine, configures middleware, and registers all route handlers. This process is handled by the `Initialize` function in the `routes` package. + +```mermaid +sequenceDiagram + participant App as "Application" + participant Router as "router.Initialize()" + participant Gin as "gin.Engine" + participant Middleware as "Middleware" + participant Handlers as "Handler Registries" + + App->>Router: Call Initialize(svcCtx) + Router->>Gin: Create new Gin engine + Router->>Gin: Set trusted proxies + Router->>Middleware: Configure recovery middleware + Router->>Middleware: Add auth middleware + Router->>Middleware: Add cache control middleware + Router->>Middleware: Add logging middleware + Router->>Handlers: Create frontend handlers + Router->>Handlers: Create render handlers + Router->>Gin: Set HTML renderer + Router->>Gin: Setup static router + Router->>Gin: Setup API router + Router->>Gin: Setup views router + Router->>Gin: Setup not found router + Router-->>App: Return configured Gin engine +``` + +Sources: [internal/routes/router.go:33-96]() + +## Middleware Components + +CSGHub's middleware components process HTTP requests before they reach route handlers, performing tasks such as authentication, logging, and error handling. + +### Middleware Chain + +The middleware chain executes in the following order: + +1. **Recovery Middleware**: Captures panics and converts them to 500 responses with error logging +2. **Authentication Middleware**: Verifies user authentication using JWT tokens +3. **Cache Control Middleware**: Manages HTTP caching headers +4. **Logging Middleware**: Logs all requests for monitoring and debugging +5. **Configuration Injection**: Injects global configuration into request contexts + +```mermaid +flowchart LR + req["Request"] --> recovery["Recovery Middleware"] + recovery --> auth["Auth Middleware"] + auth --> cache["Cache Control Middleware"] + auth --> log["Logging Middleware"] + log --> config["Config Injection"] + config --> handler["Route Handler"] +``` + +Sources: [internal/routes/router.go:61-73](), [internal/routes/router.go:98-104]() + +### Authentication Middleware + +The authentication middleware validates JWT tokens and populates user information in the request context. It's a critical component that secures routes requiring authentication. + +Key functions: +- Extracts JWT tokens from cookies or Authorization headers +- Verifies token validity using the backend service +- Attaches authenticated user information to the request context +- Handles token refreshing for extended sessions + +Sources: [internal/handlers/frontend/token.go:34-59](), [internal/routes/router.go:69]() + +## Route Groups + +CSGHub organizes routes into distinct groups based on functionality, making the codebase more maintainable and easier to navigate. + +### Static Routes + +Static routes serve frontend assets such as JavaScript, CSS, and image files directly from the filesystem. + +Sources: [internal/routes/router.go:232-250]() + +### API Routes + +API routes handle internal AJAX requests from the frontend, providing data endpoints for dynamic features. + +``` +/internal_api/ping - Health check endpoint +/internal_api/users/jwt_token - JWT token refresh +/internal_api/upload - File upload handling +``` + +Sources: [internal/routes/router.go:252-265]() + +### View Routes + +View routes render HTML templates for different parts of the application. These routes are organized into functional categories: + +| Category | Purpose | +|----------|---------| +| Error routes | Display error pages (404, 401, etc.) | +| Home routes | Main dashboard and landing pages | +| Model routes | Model repository pages | +| Dataset routes | Dataset repository pages | +| Code routes | Code repository pages | +| Space routes | Space repository pages | +| Endpoint routes | Model endpoint management | +| Finetune routes | Model fine-tuning interfaces | +| Session routes | Authentication and session management | +| Organization routes | Organization profile and management | +| Profile routes | User profile pages | +| Setting routes | User settings pages | +| Resource console routes | Resource management console | +| Admin routes | Administrative interfaces | + +Sources: [internal/routes/router.go:205-230]() + +### Resolve Routes + +Resolve routes handle repository content resolution, mapping repository paths to specific files or directories: + +``` +/:repo_type/:namespace/:name/resolve/:branch/*path +``` + +Sources: [internal/routes/router.go:259-260]() + +## Handler Registries + +The routing system relies on handler registries to organize and access route handlers. These registries group related handlers together for better code organization. + +```mermaid +classDiagram + class HandlersRegistry { + +FrontendHandlers *FrontendHandlerRegistry + +RenderHandler *RenderHandlerRegistry + +Config *config.Config + } + + class FrontendHandlerRegistry { + +PingHandler + +TokenHandler + +UploadHandler + +ResolveHandler + +SettingsHandler + ~Other handlers... + } + + class RenderHandlerRegistry { + +ErrorHandler + +ModelHandler + +DatasetHandler + +CodeHandler + +SpaceHandler + +EndpointHandler + +FinetuneHandler + +SessionHandler + ~Other handlers... + } + + HandlersRegistry --o FrontendHandlerRegistry : contains + HandlersRegistry --o RenderHandlerRegistry : contains +``` + +Sources: [internal/routes/router.go:26-31](), [internal/handlers/render/registry.go:9-26]() + +## Template Rendering + +CSGHub uses a template rendering system to generate HTML responses. The system uses the `multitemplate.Renderer` package to manage multiple templates for different views. + +The template rendering process: +1. Templates are organized into layouts and pages +2. Layouts provide common structure (header, footer, etc.) +3. Pages provide view-specific content +4. Templates are parsed from the embedded filesystem during initialization +5. Rendered templates are served in response to view route requests + +Sources: [internal/routes/router.go:106-203]() + +## Error Handling + +The routing system includes comprehensive error handling to provide appropriate responses for various error conditions: + +1. **Panic Recovery**: Catches unhandled panics, logs them with stack traces, and returns a 500 response +2. **404 Not Found**: Redirects to a dedicated not-found page for unmatched routes +3. **Authentication Errors**: Redirects unauthenticated users to appropriate error pages + +Sources: [internal/routes/router.go:61-68](), [internal/routes/router.go:267-272]() + +## JWT Authentication Flow + +JWT (JSON Web Token) authentication is central to CSGHub's authentication system. The flow begins with the authentication middleware and includes token verification and refresh mechanisms. + +```mermaid +sequenceDiagram + participant Client as "Client" + participant Router as "Router" + participant Auth as "Auth Middleware" + participant Token as "Token Handler" + participant Server as "Backend Server" + + Client->>Router: Request with JWT token + Router->>Auth: Pass request + Auth->>Server: Verify JWT token + Server-->>Auth: Return user info if valid + + alt Token valid + Auth->>Router: Continue to handler with user context + else Token invalid or expired + Auth->>Client: Return 401 Unauthorized + end + + Client->>Token: Request token refresh + Token->>Server: Create new JWT token + Server-->>Token: Return new token + Token-->>Client: Set cookie and return token +``` + +Sources: [internal/handlers/frontend/token.go:34-59](), [pkg/server/types/jwt.go:1-45]() + +## Implementation Details + +### Router Initialization + +The router is initialized in the `Initialize` function, which sets up the Gin engine, configures middleware, and registers routes: + +1. Creates a new Gin engine +2. Sets up logging +3. Configures middleware +4. Creates handler registries +5. Sets up the HTML renderer +6. Registers all routes + +Sources: [internal/routes/router.go:33-96]() + +### Global Configuration Injection + +The router injects global configuration into the request context using middleware, making configuration accessible to all handlers: + +```go +// Global configuration structure +type GlobalConfig struct { + ServerBaseUrl string + OnPremise bool + EnableHttps bool +} + +// Middleware to inject configuration +func injectConfig(config types.GlobalConfig) gin.HandlerFunc { + return func(c *gin.Context) { + c.Set("Config", config) + c.Next() + } +} +``` + +Sources: [internal/routes/router.go:98-104](), [internal/routes/router.go:206-211]() + +## Conclusion + +The routing and middleware system in CSGHub provides a structured approach to handling HTTP requests, with clear separation of concerns and organized code. The system leverages Gin's powerful routing capabilities and middleware chain to process requests efficiently and securely. + +The key strengths of the system include: +- Organized route grouping by functionality +- Comprehensive middleware chain for cross-cutting concerns +- Structured error handling and recovery +- Template-based view rendering +- JWT-based authentication + +This architecture allows CSGHub to maintain a clean codebase while providing robust HTTP request handling capabilities. \ No newline at end of file diff --git a/docs/development/21_Repository_Handlers.md b/docs/development/21_Repository_Handlers.md new file mode 100644 index 000000000..c7f84560c --- /dev/null +++ b/docs/development/21_Repository_Handlers.md @@ -0,0 +1,307 @@ +# Repository Handlers + +## Purpose and Scope + +Repository Handlers form a core component of CSGHub's backend architecture, responsible for processing HTTP requests related to different repository types (models, datasets, codes, and spaces). These handlers serve as the bridge between the router and the template rendering system, processing user interactions and generating appropriate responses. + +This document covers the structure, implementation, and flow of Repository Handlers within the CSGHub backend system. For information about the overall routing system and middleware chain, see [Routing and Middleware](#6.1). + +## Repository Handler Architecture + +The Repository Handlers in CSGHub follow a structured inheritance pattern that promotes code reuse while allowing for repository-specific customizations. + +```mermaid +classDiagram + class BaseHandler { + <> + +List(ctx *gin.Context) + +Detail(ctx *gin.Context) + +Files(ctx *gin.Context) + +Blob(ctx *gin.Context) + +Commits(ctx *gin.Context) + +Commit(ctx *gin.Context) + +NewFile(ctx *gin.Context) + +UploadFile(ctx *gin.Context) + +EditFile(ctx *gin.Context) + +Settings(ctx *gin.Context) + +Billing(ctx *gin.Context) + +Logs(ctx *gin.Context) + +Community(ctx *gin.Context) + +New(ctx *gin.Context) + } + + class BaseHandlerImpl { + -resourceType string + +List(ctx *gin.Context) + +Detail(ctx *gin.Context) + +Files(ctx *gin.Context) + +Blob(ctx *gin.Context) + +Commits(ctx *gin.Context) + +Commit(ctx *gin.Context) + +NewFile(ctx *gin.Context) + +UploadFile(ctx *gin.Context) + +EditFile(ctx *gin.Context) + +Settings(ctx *gin.Context) + +Billing(ctx *gin.Context) + +Logs(ctx *gin.Context) + +Community(ctx *gin.Context) + +New(ctx *gin.Context) + -renderShow(ctx, actionName, defaultTab) + -addResourceSpecificData(ctx, data) + } + + class ModelHandler { + <> + } + + class DatasetHandler { + <> + } + + class CodeHandler { + <> + } + + class SpaceHandler { + <> + } + + class ModelHandlerImpl { + -resourceType="models" + } + + class DatasetHandlerImpl { + -resourceType="datasets" + } + + class CodeHandlerImpl { + -resourceType="codes" + } + + class SpaceHandlerImpl { + -resourceType="spaces" + } + + BaseHandler <|-- ModelHandler + BaseHandler <|-- DatasetHandler + BaseHandler <|-- CodeHandler + BaseHandler <|-- SpaceHandler + + BaseHandlerImpl <|-- ModelHandlerImpl + BaseHandlerImpl <|-- DatasetHandlerImpl + BaseHandlerImpl <|-- CodeHandlerImpl + BaseHandlerImpl <|-- SpaceHandlerImpl + + ModelHandler <|.. ModelHandlerImpl + DatasetHandler <|.. DatasetHandlerImpl + CodeHandler <|.. CodeHandlerImpl + SpaceHandler <|.. SpaceHandlerImpl +``` + +Sources: [internal/handlers/render/repo.go:39-53](), [internal/handlers/render/models.go](), [internal/handlers/render/datasets.go](), [internal/handlers/render/codes.go](), [internal/handlers/render/spaces.go]() + +## Repository Types and Handler Implementation + +CSGHub supports four primary repository types, each with its dedicated handler implementation: + +| Repository Type | Handler Interface | Implementation | Resource Type Value | +|-----------------|-------------------|----------------|---------------------| +| Models | `ModelHandler` | `ModelHandlerImpl` | "models" | +| Datasets | `DatasetHandler` | `DatasetHandlerImpl` | "datasets" | +| Codes | `CodeHandler` | `CodeHandlerImpl` | "codes" | +| Spaces | `SpaceHandler` | `SpaceHandlerImpl` | "spaces" | + +Each specific handler implementation primarily differs in the `resourceType` field value, which is used to determine which templates to render and what type-specific data to include. + +Sources: [internal/handlers/render/models.go](), [internal/handlers/render/datasets.go](), [internal/handlers/render/codes.go](), [internal/handlers/render/spaces.go]() + +## Route to Handler Mapping + +The system maps HTTP routes to handler methods based on the repository type, following a consistent pattern across all repository types: + +```mermaid +flowchart LR + Router["Gin Router"] --> ModelRoutes["models/:namespace/:model_name"] + Router --> DatasetRoutes["datasets/:namespace/:dataset_name"] + Router --> CodeRoutes["codes/:namespace/:code_name"] + Router --> SpaceRoutes["spaces/:namespace/:space_name"] + + subgraph "Unauthenticated Routes" + ModelRoutes --> MLH["ModelHandler.List()"] + ModelRoutes --> MDH["ModelHandler.Detail()"] + ModelRoutes --> MFH["ModelHandler.Files()"] + ModelRoutes --> MBH["ModelHandler.Blob()"] + ModelRoutes --> MCH["ModelHandler.Commits()"] + ModelRoutes --> MCOH["ModelHandler.Community()"] + end + + subgraph "Authenticated Routes" + ModelRoutes --> MNH["ModelHandler.New()"] + ModelRoutes --> MNFH["ModelHandler.NewFile()"] + ModelRoutes --> MUFH["ModelHandler.UploadFile()"] + ModelRoutes --> MEFH["ModelHandler.EditFile()"] + ModelRoutes --> MSH["ModelHandler.Settings()"] + end +``` + +Sources: [internal/routes/models.go](), [internal/routes/datasets.go](), [internal/routes/codes.go](), [internal/routes/spaces.go]() + +## Handler Method Behavior + +The `BaseHandlerImpl` provides a default implementation for all handler methods, with consistent behavior: + +| Method | Purpose | URL Pattern Example | Template Name Pattern | +|--------|---------|---------------------|------------------------| +| `List` | List repositories | `/models` | `{resourceType}_index` | +| `Detail` | Show repo details | `/models/:namespace/:model_name` | `{resourceType}_show` (summary tab) | +| `Files` | Display file tree | `.../files/:branch/*path` | `{resourceType}_show` (files tab) | +| `Blob` | Show file content | `.../blob/:branch/*path` | `{resourceType}_show` (files tab) | +| `Commits` | Show commit history | `.../commits` | `{resourceType}_show` (files tab) | +| `Commit` | Show commit details | `.../commit/:commit_id` | `{resourceType}_show` (files tab) | +| `NewFile` | Create file form | `.../:branch/new` | `{resourceType}_show` (files tab) | +| `UploadFile` | Upload file form | `.../:branch/upload` | `{resourceType}_show` (files tab) | +| `EditFile` | Edit file form | `.../edit/:branch/*path` | `{resourceType}_show` (files tab) | +| `Settings` | Repository settings | `.../settings` | `{resourceType}_show` (settings tab) | +| `Billing` | Billing information | `.../billing` | `{resourceType}_show` (billing tab) | +| `Community` | Community info | `.../community` | `{resourceType}_show` (community tab) | +| `New` | New repo form | `/models/new` | `{resourceType}_new` | + +Most methods call a common `renderShow` helper method that prepares a data context and renders the appropriate template. + +Sources: [internal/handlers/render/repo.go:68-123]() + +## Request Processing Flow + +The following diagram illustrates how a request flows through the Repository Handlers system: + +```mermaid +sequenceDiagram + participant Client + participant GinRouter as "Gin Router" + participant AuthMiddleware as "Authentication Middleware" + participant RepoHandler as "Repository Handler" + participant RenderBase as "RenderBase.RenderTemplate()" + + Client->>GinRouter: HTTP Request + Note over GinRouter: Routes mapped in routes/*.go + + alt Authenticated Route + GinRouter->>AuthMiddleware: middleware.Instance.CheckCurrentUser() + AuthMiddleware->>RepoHandler: If authenticated, proceed + else Public Route + GinRouter->>RepoHandler: Direct to handler + end + + RepoHandler->>RepoHandler: Process request + + alt Detail/Show Views + RepoHandler->>RepoHandler: renderShow(ctx, actionName, defaultTab) + RepoHandler->>RepoHandler: addResourceSpecificData(ctx, data) + else List/New Views + RepoHandler->>RenderBase: RenderTemplate(ctx, template, data) + end + + RenderBase-->>Client: HTML Response +``` + +Sources: [internal/routes/models.go](), [internal/handlers/render/repo.go:125-163]() + +## Template Rendering and Data Context + +The `renderShow` method plays a key role in preparing the data context for template rendering: + +1. It creates a base data map with common information: + ```go + data := map[string]interface{}{ + "namespace": ctx.Param("namespace"), + "actionName": actionName, + "currentPath": strings.TrimPrefix(ctx.Param("path"), "/"), + "currentBranch": ctx.Param("branch"), + "defaultTab": defaultTab, + } + ``` + +2. It adds resource-specific data through `addResourceSpecificData`: + ```go + switch b.resourceType { + case "datasets": + data["datasetName"] = ctx.Param("dataset_name") + case "models": + data["modelName"] = ctx.Param("model_name") + case "codes": + data["codeName"] = ctx.Param("code_name") + case "spaces": + data["spaceName"] = ctx.Param("space_name") + // ...others... + } + ``` + +3. It renders the template with the prepared data: + ```go + RenderBaseInstance.RenderTemplate(ctx, b.resourceType+"_show", data) + ``` + +Sources: [internal/handlers/render/repo.go:125-143](), [internal/handlers/render/repo.go:145-163]() + +## License Handling + +The Repository Handlers system also includes built-in support for managing licenses: + +```go +var DefaultLicenses = [][]string{ + {"apache-2.0", "Apache-2.0"}, + {"mit", "MIT"}, + {"lgpl", "LGPL"}, + // ... additional licenses ... +} +``` + +This licensing information is used when creating new repositories and is passed to the template rendering system as a JSON string. + +Sources: [internal/handlers/render/repo.go:12-37](), [internal/handlers/render/repo.go:122]() + +## Extension Points + +While the Repository Handlers system follows a common pattern, it includes several extension points: + +1. **Resource Type Customization**: Each handler implementation sets a specific `resourceType` value +2. **Method Overriding**: Specific handlers can override any base method to provide custom behavior +3. **Template Mapping**: The template naming convention (`{resourceType}_show`) allows for resource-specific templates + +Sources: [internal/handlers/render/models.go:11-17](), [internal/handlers/render/datasets.go:11-17](), [internal/handlers/render/codes.go:11-17](), [internal/handlers/render/spaces.go:11-17]() + +## Repository Handler Registration + +Repository handlers must be registered in the `HandlersRegistry` for use by the router: + +```mermaid +flowchart TD + HandlersRegistry["HandlersRegistry"] --> RenderHandler["RenderHandler"] + RenderHandler --> ModelHandler["ModelHandler"] + RenderHandler --> DatasetHandler["DatasetHandler"] + RenderHandler --> CodeHandler["CodeHandler"] + RenderHandler --> SpaceHandler["SpaceHandler"] + + Router["Gin Router"] --> ModelRoutes["registerModelRoutes()"] + Router --> DatasetRoutes["registerDatasetRoutes()"] + Router --> CodeRoutes["registerCodeRoutes()"] + Router --> SpaceRoutes["registerSpaceRoutes()"] + + ModelRoutes --> |"handlers.RenderHandler.ModelHandler"| ModelHandler + DatasetRoutes --> |"handlers.RenderHandler.DatasetHandler"| DatasetHandler + CodeRoutes --> |"handlers.RenderHandler.CodeHandler"| CodeHandler + SpaceRoutes --> |"handlers.RenderHandler.SpaceHandler"| SpaceHandler +``` + +Sources: [internal/routes/models.go:8-10](), [internal/routes/datasets.go:8-10](), [internal/routes/codes.go:8-10](), [internal/routes/spaces.go:8-10]() + +## Summary + +Repository Handlers in CSGHub form a well-structured system based on a common interface and implementation pattern. This approach enables code reuse while allowing for repository-specific customizations through: + +1. A shared `BaseHandler` interface defining all handler methods +2. A common `BaseHandlerImpl` providing default implementations +3. Specialized handlers for each repository type (models, datasets, codes, spaces) +4. Resource-specific data handling in templates + +This architecture makes it easy to add new repository types or customize existing behavior while maintaining a consistent user experience across the platform. \ No newline at end of file diff --git a/docs/development/22_Development_and_Deployment.md b/docs/development/22_Development_and_Deployment.md new file mode 100644 index 000000000..f06cdc13a --- /dev/null +++ b/docs/development/22_Development_and_Deployment.md @@ -0,0 +1,317 @@ +# Development and Deployment + +This document provides a comprehensive guide to the development workflow, build process, and deployment procedures for the CSGHub platform. It covers setting up a development environment, building the application components, and deploying through CI/CD pipelines. + +## Development Environment + +### Prerequisites + +To develop for CSGHub, you need the following tools installed: + +- Node.js and Yarn for frontend development +- Go 1.23.3 or later for backend development +- Docker for containerization and testing +- Git for version control + +### Frontend Development + +The frontend is built with Vue.js 3 and uses Vite as the build tool. The following npm scripts are defined in the package.json file: + +``` +yarn dev # Starts the development server with hot reloading +yarn dev_build # Builds in development mode with watch enabled +yarn test # Runs Vitest tests +yarn coverage # Runs tests with coverage reporting +yarn build # Creates a production build +yarn preview # Previews the production build locally +``` + +Sources: [frontend/package.json:6-13]() + +### Backend Development + +The backend is written in Go using the Gin web framework. The main entry point is `cmd/csghub-portal`. The Go dependencies are managed using Go modules. + +Sources: [go.mod:1-3](), [Dockerfile:14]() + +## Build Configuration + +### Frontend Build Configuration + +CSGHub uses different Vite configurations for development and production environments. + +#### Development Build Config + +```mermaid +graph TD + subgraph "Development Build Configuration" + dev["DEV_CONFIG"] --> plugins["Vue Plugin"] + dev --> build[Build Options] + build --> ro["rollupOptions"] + build --> opts["Development Options"] + opts --> |"treeshake: false"| opt1["Faster Builds"] + opts --> |"sourcemap: true"| opt2["Debugging"] + opts --> |"minify: false"| opt3["Quick Rebuilds"] + opts --> |"emptyOutDir: false"| opt4["Preserve Files"] + ro --> chunks["Manual Chunk Strategy"] + end +``` + +The development build focuses on developer experience with: +- Disabled tree-shaking for faster builds +- Source maps for debugging +- No minification for quicker rebuilds +- Watch mode for immediate feedback +- Manual chunk strategy for common libraries + +Sources: [frontend/vite.config.js:32-76]() + +#### Production Build Config + +```mermaid +graph TD + subgraph "Production Build Configuration" + prod["PROD_CONFIG"] --> plugins["Vue Plugin"] + prod --> build[Build Options] + build --> ro["rollupOptions"] + build --> opts["Production Options"] + opts --> |"emptyOutDir: true"| opt1["Clean Output"] + opts --> |"minify: terser"| opt2["Optimized Code"] + opts --> |"drop_console: true"| opt3["Remove Console Logs"] + opts --> |"drop_debugger: true"| opt4["Remove debugger"] + ro --> chunks["Manual Chunk Strategy"] + end +``` + +The production build optimizes for end-user performance: +- Terser minification for smaller bundle size +- Removal of console logs and debugger statements +- Clean output directory +- Optimized chunk splitting for better loading performance + +Sources: [frontend/vite.config.js:78-118]() + +## Build Process + +### Frontend Build Process + +The frontend build process is managed by Vite, which compiles and bundles Vue components, JavaScript, CSS, and other assets. + +```mermaid +flowchart TD + src["Source Files"] --> vite["Vite Build System"] + vite --> bundling["Bundle & Transform"] + bundling --> chunks["Create Chunks"] + chunks --> |"vendor.js"| vendor["Third-party Libs"] + chunks --> |"components.js"| comp["Component Code"] + chunks --> |"lodash.js"| lodash["Lodash Utils"] + chunks --> |"element-plus.js"| ep["Element Plus UI"] + chunks --> |"highlightjs.js"| hl["Syntax Highlighting"] + chunks --> assets["Asset Processing"] + vendor --> dist["dist/ Directory"] + comp --> dist + lodash --> dist + ep --> dist + hl --> dist + assets --> dist +``` + +Vite is configured to split the code into logical chunks: +- Vendor chunks for third-party libraries +- Component chunks for reusable UI components +- Specialized chunks for large libraries (lodash, element-plus, highlight.js) + +Sources: [frontend/vite.config.js:84-101]() + +### Backend Build Process + +The backend is built using Go's standard build toolchain. The Dockerfile shows the build process: + +```mermaid +flowchart LR + goFiles["Go Source Files"] --> goMod["go mod tidy"] + goMod --> goBuild["go build"] + goBuild --> |"Target Platform"| binary["csghub-portal Binary"] + + subgraph "Cross-Platform Build" + env["Set GOOS/GOARCH"] --> crossBuild["Platform-specific Build"] + end + + goBuild --> env +``` + +The Go build process includes: +1. Dependency resolution via `go mod tidy` +2. Compilation with platform-specific settings +3. Generation of the `csghub-portal` binary + +Sources: [Dockerfile:9-14]() + +## Docker Build Process + +CSGHub uses a multi-stage Docker build to create a lean production image: + +```mermaid +flowchart TD + subgraph "Builder Stage" + src["Source Code"] --> goMod["go mod tidy"] + src --> npmIn["yarn install"] + npmIn --> npmBuild["yarn build"] + goMod --> goBuild["go build"] + npmBuild --> artifacts["Frontend Artifacts"] + goBuild --> binary["Backend Binary"] + end + + subgraph "Final Stage" + baseImage["bitnami/minideb"] --> install["Install CA Certificates"] + install --> copy["Copy Binary from Builder"] + binary --> copy + end +``` + +The Dockerfile uses: +1. A builder stage with Go and Node.js for compiling both frontend and backend +2. A minimal final stage with only the compiled binary +3. Platform-specific build flags for cross-architecture support + +Sources: [Dockerfile:1-19]() + +## Testing + +### Frontend Testing + +The frontend uses Vitest for unit testing with the following commands: + +- `yarn test`: Run all tests in watch mode +- `yarn coverage`: Generate test coverage reports + +Sources: [frontend/package.json:9-10]() + +### Backend Testing + +Backend testing uses Go's standard testing package. Tests can be run with: + +```bash +go test ./... +``` + +## Deployment + +### CI/CD Pipeline + +CSGHub uses GitLab CI/CD for automated builds and deployments. The pipeline consists of three stages: + +```mermaid +flowchart TD + subgraph "GitLab CI/CD Pipeline" + push["Git Push"] --> condition{"Branch/Tag Check"} + condition -->|"csghub-staging branch"| staging["Staging Pipeline"] + condition -->|"v*.*.* tag"| prod["Production Pipeline"] + + subgraph "Build Stage" + buildx["Setup Docker Buildx"] --> login["Registry Login"] + login --> build["Multi-architecture Build"] + build --> push["Push to Registry"] + end + + staging --> buildx + prod --> buildx + + push --> deploy["Deploy Stage"] + deploy -->|"Connect to Server"| server["Remote Server"] + server --> pull["Pull Image"] + pull --> recreate["Recreate Container"] + end +``` + +The pipeline is triggered by: +- Pushes to the `csghub-staging` branch for staging deployments +- Tags matching the pattern `v*.*.*` for production releases + +Sources: [.gitlab-ci.yml:9-12](), [.gitlab-ci.yml:15-21]() + +### Build Stage + +The build stage creates multi-architecture Docker images: + +1. Sets up Docker Buildx for multi-platform builds +2. Authenticates with the GitLab container registry +3. Builds images for both amd64 and arm64 architectures +4. Pushes images to the registry + +Key configuration: +- Platform targets: `linux/amd64,linux/arm64` +- Image tag based on branch/tag name +- No default attestations + +Sources: [.gitlab-ci.yml:23-44]() + +### Deploy Stage + +The deployment process: + +1. Connects to the deployment server via SSH +2. Sets environment variables for the deployment +3. Pulls the latest image from the registry +4. Updates the running container using docker-compose +5. Cleans up unused images + +Sources: [.gitlab-ci.yml:45-69]() + +## Configuration + +CSGHub is configured through environment variables, which are loaded and processed by the `envconfig` package. + +Key configuration categories: + +| Category | Environment Variables | Description | +|----------|----------------------|-------------| +| Server | `CSGHUB_PORTAL_SERVER_PORT`, `CSGHUB_PORTAL_ENABLE_HTTPS` | Basic server configuration | +| Starhub | `CSGHUB_PORTAL_STARHUB_BASE_URL`, `CSGHUB_PORTAL_STARHUB_API_KEY` | Connection to Starhub server | +| Database | `CSGHUB_PORTAL_DATABASE_DSN`, `CSGHUB_PORTAL_DATABASE_DIALECT` | Database connection settings | +| S3 Storage | `CSGHUB_PORTAL_S3_ENDPOINT`, `CSGHUB_PORTAL_S3_ACCESS_KEY_ID` | Object storage configuration | + +Sources: [config/config.go:14-41]() + +### Configuration Loading + +The configuration is loaded at application startup using the `LoadConfig` function: + +```mermaid +flowchart LR + start["Application Start"] --> loadConfig["LoadConfig()"] + loadConfig --> envconfig["envconfig.Process()"] + envconfig --> |"Environment Variables"| config["Config Struct"] + config --> app["Application"] +``` + +Sources: [config/config.go:43-50]() + +## Troubleshooting + +Common issues and solutions: + +1. **Frontend build errors**: Verify Node.js/Yarn versions and clear node_modules. +2. **Cross-platform issues**: Ensure Docker buildx is configured correctly for multi-architecture builds. +3. **Deployment failures**: Check SSH access and environment variables on deployment servers. + +## Bug Reporting + +When reporting bugs, please use the bug report template and include: + +- CSGHub version +- Operating system +- Hardware configuration +- Deployment method (docker compose, helm chart) +- Steps to reproduce + +Sources: [.github/ISSUE_TEMPLATE/bug_report.md:12-36]() + +## Feature Requests + +Feature enhancement requests should: +- Clearly describe the proposed solution +- Explain the benefits of the feature +- Include links to relevant documentation or design documents + +Sources: [.github/ISSUE_TEMPLATE/feature_request.md:12-21]() \ No newline at end of file diff --git a/docs/development/2_Repository_Management_System.md b/docs/development/2_Repository_Management_System.md new file mode 100644 index 000000000..855604ba7 --- /dev/null +++ b/docs/development/2_Repository_Management_System.md @@ -0,0 +1,494 @@ +# Repository Management System + +The Repository Management System is the core framework in CSGHub that manages different types of repositories (models, datasets, code, spaces, endpoints, finetunes, and collections). It provides a unified set of components and APIs to display, browse, filter, and interact with repositories while supporting type-specific behaviors for each repository kind. + +This document explains the overall architecture, components, and interactions of the repository system. For information on specific repository types and their specialized capabilities, see [Model Deployment and Fine-tuning](#3) or [User and Authentication System](#4). + +## System Overview + +The Repository Management System handles several repository types, each with common functionality but also specialized features: + +| Repository Type | Purpose | Key Features | +|-----------------|---------|--------------| +| Model | ML models | Inference, fine-tuning, evaluation | +| Dataset | Training data | Data previews, versioning | +| Code | Development code | Code browsing, versioning | +| Space | Application deployments | Runtime environments, user interfaces | +| Endpoint | Inference services | Runtime options, scaling, monitoring | +| Finetune | Fine-tuned models | Training status, parameters | +| Collection | Repository grouping | Themed collections, multiple repos | + +The system provides consistent interfaces for all repository types while enabling specialized functionality for each. + +### Architecture Overview + +```mermaid +graph TD + User["User"] --> |"Interacts with"| UI["Repository UI Components"] + UI --> |"Uses"| RepoDetailStore["RepoDetailStore.js"] + UI --> |"Makes API calls to"| Backend["Backend API Routes"] + Backend --> |"Retrieves/updates"| DB[("Database")] + Backend --> |"Manages"| Storage["File Storage"] + + subgraph "Frontend Components" + UI --> RepoHeader["RepoHeader.vue"] + UI --> RepoCards["RepoCards.vue"] + UI --> RepoTabs["Repository Detail Tabs"] + RepoDetailStore + end + + subgraph "Backend Routes" + Backend --> ModelsAPI["models.go"] + Backend --> DatasetsAPI["datasets.go"] + Backend --> CodesAPI["codes.go"] + Backend --> SpacesAPI["spaces.go"] + end +``` + +Sources: +- [frontend/src/components/shared/RepoHeader.vue:1-368]() +- [frontend/src/components/shared/RepoCards.vue:1-332]() +- [frontend/src/stores/RepoDetailStore.js:1-245]() +- [internal/routes/models.go:1-31]() +- [internal/routes/datasets.go:1-31]() +- [internal/routes/codes.go:1-31]() +- [internal/routes/spaces.go:1-32]() + +## Repository Data Model + +All repository types share a common data model while having type-specific extensions. The state for repositories is centrally managed through the `RepoDetailStore`. + +### Core Repository Properties + +```mermaid +classDiagram + class RepoDetailStore { + +int id + +string name + +string nickname + +string description + +string path + +bool privateVisibility + +int likes + +array tags + +object user + +string license + +int downloads + +string status + +string repoType + +bool isPrivate() + +updateVisibility() + +updateLikes() + +updateUserLikes() + +initialize() + } +``` + +The `RepoDetailStore` contains all properties for the currently viewed repository. It includes common properties shared by all repository types as well as type-specific properties that only apply to certain repository types. + +Sources: +- [frontend/src/stores/RepoDetailStore.js:6-243]() + +### Type-Specific Extensions + +```mermaid +graph TD + RepoDetail["RepoDetailStore"] --> CommonProps["Common Properties:
id, name, description, etc."] + RepoDetail --> TypeProps["Type-Specific Properties"] + + TypeProps --> ModelProps["Model Properties:
enableInference,
enableFinetune, etc."] + TypeProps --> EndpointProps["Endpoint Properties:
endpoint, actualReplica,
deployId, etc."] + TypeProps --> SpaceProps["Space Properties:
sdk, hardware,
coverImageUrl, etc."] + TypeProps --> FinetuneProps["Finetune Properties:
modelId, runtimeFramework,
deployName, etc."] + TypeProps --> CollectionProps["Collection Properties:
avatar, theme,
repositories, etc."] +``` + +The store includes specialized properties for each repository type: + +```javascript +// For models +enableInference: true/false +enableFinetune: true/false +enableEvaluation: true/false + +// For endpoints +endpoint: "endpoint-url" +minReplica: 1 +maxReplica: 3 +actualReplica: 2 +proxyEndpoint: "proxy-url" +deployId: 123 + +// For spaces +sdk: "gradio" +hardware: "cpu" +coverImageUrl: "image-url" + +// For collections +avatar: "avatar-url" +theme: "#F5F3FF" +repositories: [] +``` + +Sources: +- [frontend/src/stores/RepoDetailStore.js:9-236]() +- [frontend/src/components/finetune/FinetuneDetail.vue:5-16]() +- [frontend/src/components/endpoints/EndpointDetail.vue:4-16]() +- [frontend/src/components/collections/CollectionsDetail.vue:6-17]() + +## User Interface Components + +The Repository Management System includes several UI components that collectively provide the repository interface. + +### Repository Header + +The `RepoHeader` component displays key repository information at the top of repository pages: + +```mermaid +graph LR + RepoHeader["RepoHeader.vue"] --> Display["Repository Display"] + Display --> RepoName["Repository Name/Nickname"] + Display --> RepoPath["Owner/Path"] + Display --> RepoType["Type-Specific Header"] + Display --> RepoActions["Repository Actions"] + + RepoType --> ModelHeader["Model Header"] + RepoType --> DatasetHeader["Dataset Header"] + RepoType --> CodeHeader["Code Header"] + RepoType --> SpaceHeader["Space Header"] + RepoType --> EndpointHeader["Endpoint Header"] + RepoType --> FinetuneHeader["Finetune Header"] + + RepoActions --> LikeButton["Like Button"] + RepoActions --> CopyPath["Copy Path"] + RepoActions --> PrivateBadge["Private Badge"] + RepoActions --> StatusIndicator["Status Indicator"] +``` + +The header adapts its display based on repository type: +- For datasets: Shows a dataset icon and name +- For endpoints: Shows status and resource information +- For finetunes: Shows finetune status and model information +- For other types: Shows repository type icon and metadata + +Sources: +- [frontend/src/components/shared/RepoHeader.vue:1-368]() +- [frontend/src/components/__tests__/shared/RepoHeader.spec.js:1-332]() + +### Repository Cards Browser + +The `RepoCards` component provides a browsing interface for repositories: + +```mermaid +graph TD + RepoCards["RepoCards.vue"] --> FilteringOptions["Filtering Options"] + RepoCards --> CardGrid["Repository Cards Grid"] + RepoCards --> PaginationControl["Pagination Controls"] + + FilteringOptions --> TextSearch["Name Search"] + FilteringOptions --> SortDropdown["Sort Dropdown"] + FilteringOptions --> TagFiltering["Tag Filtering"] + FilteringOptions --> SourceFilter["Source Filtering"] + FilteringOptions --> SDKFilter["SDK Filter (Spaces)"] + + CardGrid --> RepoItem["RepoItem Component"] + CardGrid --> SpaceItem["AppSpaceItem Component"] + + TagFiltering --> TagSidebar["TagSidebar Component"] +``` + +`RepoCards` supports filtering by: +- Text search by name +- Sorting (trending, recently updated, downloads, favorites) +- Tags via the sidebar +- Source (for on-premise installations) +- SDK type (for spaces) + +Sources: +- [frontend/src/components/shared/RepoCards.vue:1-332]() + +### Repository Detail Views + +Each repository type has a specialized detail view that extends a common pattern: + +```mermaid +graph TD + DetailView["Repository Detail View"] --> RepoHeader["RepoHeader"] + DetailView --> RepoTabs["Repository Tabs"] + + RepoTabs --> MainTab["Main Content Tab"] + RepoTabs --> FilesTab["Files Tab"] + RepoTabs --> CommitsTab["Commits Tab"] + RepoTabs --> SettingsTab["Settings Tab"] + + MainTab --> ModelSettings["Model Settings"] + MainTab --> DatasetViewer["Dataset Viewer"] + MainTab --> EndpointPlayground["Endpoint Playground"] + MainTab --> FinetuneMonitor["Finetune Monitor"] + MainTab --> CollectionsList["Collections List"] +``` + +The detail views follow a consistent pattern but adapt to each repository type: +- Model detail: Shows model metadata, cards +- Dataset detail: Shows dataset preview, metadata +- Endpoint detail: Shows endpoint status, playground +- Finetune detail: Shows training status, settings +- Collection detail: Shows contained repositories + +Sources: +- [frontend/src/components/finetune/FinetuneDetail.vue:1-325]() +- [frontend/src/components/endpoints/EndpointDetail.vue:1-225]() +- [frontend/src/components/collections/CollectionsDetail.vue:1-173]() + +## Repository Operations + +The Repository Management System supports various operations on repositories. + +### Repository Browsing and Filtering + +Users can browse repositories with filtering and sorting options: + +```mermaid +sequenceDiagram + actor User + participant Browser + participant API as "Backend API" + + User->>Browser: Navigate to repository list + Browser->>API: GET /models?sort=trending&search=text + Note over API: Query parameters control filtering + API->>Browser: Return filtered repositories + Browser->>User: Display repository cards + + User->>Browser: Change filters + Browser->>API: GET /models?sort=recently_update&tag_name=nlp + API->>Browser: Return newly filtered results + Browser->>User: Update repository display +``` + +Key filtering options include: +- Text search via `search` parameter +- Sorting via `sort` parameter (trending, recently_update, most_download, most_favorite) +- Tag filtering via `tag_category` and `tag_name` parameters +- Source filtering via `source` parameter +- SDK filtering via `sdk` parameter (for spaces) +- Pagination via `page` and `per` parameters + +Sources: +- [frontend/src/components/shared/RepoCards.vue:254-308]() +- [frontend/src/locales/en_js/models.js:1-75]() +- [frontend/src/locales/zh_js/models.js:1-72]() + +### Repository Detail Display + +When viewing a specific repository, the system follows this pattern: + +```mermaid +sequenceDiagram + actor User + participant Browser + participant RepoStore as "RepoDetailStore" + participant API as "Backend API" + + User->>Browser: Navigate to repository detail + Browser->>API: GET /models/{namespace}/{name} + API->>Browser: Return repository data + Browser->>RepoStore: initialize(data, 'model') + RepoStore->>Browser: Update UI with repository data + Browser->>User: Show repository detail view + + User->>Browser: Click like button + Browser->>API: PUT /user/{name}/likes/{repoId} + API->>Browser: Return success + Browser->>RepoStore: updateLikes(newCount) + Browser->>RepoStore: updateUserLikes(true) + RepoStore->>Browser: Update UI with new like count + Browser->>User: Show updated like count +``` + +Sources: +- [frontend/src/components/shared/RepoHeader.vue:307-335]() +- [frontend/src/components/finetune/FinetuneDetail.vue:195-212]() +- [frontend/src/components/endpoints/EndpointDetail.vue:129-144]() +- [frontend/src/components/collections/CollectionsDetail.vue:153-162]() + +### Repository Empty States + +The system also handles empty states, such as when no models are available: + +```mermaid +graph TD + EmptyState["Empty State Detection"] --> CheckCount["Check Repository Count"] + CheckCount --> |"Count = 0"| ShowEmptyUI["Show Empty UI"] + CheckCount --> |"Count > 0"| ShowNormalUI["Show Normal Repository List"] + + ShowEmptyUI --> EmptyModels["EmptyModels Component"] + EmptyModels --> UploadOption["Upload Local Model Button"] + EmptyModels --> SyncOption["Multi-Sync Button"] +``` + +The empty state provides guidance to users on how to add their first repositories. + +Sources: +- [frontend/src/components/shared/RepoCards.vue:113-115]() +- [frontend/src/components/models/EmptyModels.vue:1-15]() + +## Backend Integration + +The repository frontend interfaces with backend routes to retrieve and manipulate data. + +### Backend Routes Structure + +```mermaid +graph TD + GinEngine["Gin Router Engine"] --> ModelRoutes["Model Routes"] + GinEngine --> DatasetRoutes["Dataset Routes"] + GinEngine --> CodeRoutes["Code Routes"] + GinEngine --> SpaceRoutes["Space Routes"] + + ModelRoutes --> ListModels["GET /models"] + ModelRoutes --> ModelDetail["GET /models/:namespace/:model_name"] + ModelRoutes --> ModelFiles["GET /models/:namespace/:model_name/files/:branch/*path"] + ModelRoutes --> ModelSettings["GET /models/:namespace/:model_name/settings"] + + DatasetRoutes --> ListDatasets["GET /datasets"] + DatasetRoutes --> DatasetDetail["GET /datasets/:namespace/:dataset_name"] + DatasetRoutes --> DatasetFiles["GET /datasets/:namespace/:dataset_name/files/:branch/*path"] + DatasetRoutes --> DatasetSettings["GET /datasets/:namespace/:dataset_name/settings"] + + %% Similar pattern repeated for other repository types +``` + +The backend implements consistent route patterns across repository types: +- List route: `GET /{repoType}s` +- Detail route: `GET /{repoType}s/:namespace/:repo_name` +- Files route: `GET /{repoType}s/:namespace/:repo_name/files/:branch/*path` +- Settings route: `GET /{repoType}s/:namespace/:repo_name/settings` + +Sources: +- [internal/routes/models.go:8-31]() +- [internal/routes/datasets.go:8-31]() +- [internal/routes/codes.go:8-31]() +- [internal/routes/spaces.go:8-32]() +- [internal/handlers/render/repo.go:38-163]() + +### Base Handler Implementation + +The backend uses a common base handler implementation for all repository types: + +```mermaid +classDiagram + class BaseHandler { + +List(ctx) + +Detail(ctx) + +Files(ctx) + +Blob(ctx) + +Commits(ctx) + +Commit(ctx) + +NewFile(ctx) + +UploadFile(ctx) + +EditFile(ctx) + +Settings(ctx) + +Community(ctx) + +New(ctx) + } + + BaseHandler <|-- ModelHandler + BaseHandler <|-- DatasetHandler + BaseHandler <|-- CodeHandler + BaseHandler <|-- SpaceHandler + + class BaseHandlerImpl { + +string resourceType + +List(ctx) + +Detail(ctx) + +Files(ctx) + +renderShow(ctx, action, tab) + } + + class ModelHandlerImpl { + +resourceType = "models" + } + + class DatasetHandlerImpl { + +resourceType = "datasets" + } +``` + +This design provides consistent behaviors across repository types while allowing for type-specific customizations. + +Sources: +- [internal/handlers/render/repo.go:39-53]() +- [internal/handlers/render/models.go:3-17]() +- [internal/handlers/render/datasets.go:3-17]() +- [internal/handlers/render/codes.go:3-17]() +- [internal/handlers/render/spaces.go:3-17]() + +## State Management + +The Repository Management System uses Pinia for state management, with the `RepoDetailStore` as the central store. + +### Store Initialization and Persistence + +```mermaid +sequenceDiagram + participant Component as "Vue Component" + participant Store as "RepoDetailStore" + participant Pinia as "Pinia (persist plugin)" + participant API as "Backend API" + + Component->>API: Fetch repository data + API->>Component: Return repository data + Component->>Store: initialize(data, repoType) + + Store->>Store: Set all repository properties + Store->>Pinia: Save to persistent storage + + Note over Component,Store: Later, when revisiting the page + + Component->>Store: Check if initialized and same repo + Store->>Pinia: Load persisted data + Store->>Component: Use cached data +``` + +The store is initialized when a user navigates to a repository detail page. The store: +1. Sets the repository type and initialized flag +2. Populates common and type-specific properties +3. Uses Pinia's persistence plugin to cache data + +Sources: +- [frontend/src/stores/RepoDetailStore.js:6-243]() +- [frontend/src/components/finetune/FinetuneDetail.vue:283-290]() +- [frontend/src/components/endpoints/EndpointDetail.vue:195-197]() +- [frontend/src/components/collections/CollectionsDetail.vue:167-169]() + +### API Integration for State Updates + +The repository components integrate with backend APIs to update repository state: + +```mermaid +sequenceDiagram + participant Component as "Vue Component" + participant Store as "RepoDetailStore" + participant API as "Backend API" + + Component->>Component: User action (e.g., like) + Component->>API: PUT /user/{name}/likes/{repoId} + API->>Component: Return success + Component->>Store: updateLikes(store.likes + 1) + Component->>Store: updateUserLikes(true) + Store->>Component: Update UI with new state +``` + +This pattern ensures that UI state stays synchronized with server state. + +Sources: +- [frontend/src/components/shared/RepoHeader.vue:307-335]() + +## Conclusion + +The Repository Management System provides a unified framework for handling different repository types in CSGHub. It uses consistent patterns for displaying, browsing, and manipulating repositories while accommodating type-specific behaviors. The system is built on a combination of Vue.js components, Pinia state management, and RESTful backend routes. + +This architecture enables CSGHub to support diverse repository types (models, datasets, code, spaces, endpoints, finetunes, and collections) through shared components and patterns while allowing for specialized functionality where needed. + +For more detailed information on specific repository types, see [Model Deployment and Fine-tuning](#3) or [Repository Detail Components](#2.1). \ No newline at end of file diff --git a/docs/development/3_Repository_Detail_Components.md b/docs/development/3_Repository_Detail_Components.md new file mode 100644 index 000000000..b45c4405a --- /dev/null +++ b/docs/development/3_Repository_Detail_Components.md @@ -0,0 +1,366 @@ +# Repository Detail Components + +This document details the components used for displaying repository information and details in CSGHub. These components form the core user interface for viewing and interacting with repositories of various types (models, datasets, endpoints, finetunes, and collections). For information about repository browsing and filtering, see [Repository Browsing and Filtering](#2.3), and for information about cloning and other actions, see [Repository Clone and Actions](#2.2). + +## Overview + +The Repository Detail Components system provides a consistent interface for displaying repository information while accommodating the specific needs of different repository types. The system consists of a header component, backing store, and content display components that adapt based on the repository type. + +```mermaid +graph TD + subgraph "Repository Detail Components" + RH["RepoHeader"] + RS["RepoSummary"] + RT["RepoTabs"] + RDS["RepoDetailStore"] + + RH --> RDS + RS --> RDS + RT --> RDS + end + + subgraph "Repository Types" + Model["Model Repository"] + Dataset["Dataset Repository"] + Endpoint["Endpoint"] + Finetune["Finetune"] + Collection["Collection"] + end + + Model --> RH + Dataset --> RH + Endpoint --> RH + Finetune --> RH + Collection --> RH + + Model --> RS + Dataset --> RS + + Model --> RT + Endpoint --> RT + Finetune --> RT + Collection --> RT +``` + +Sources: [frontend/src/components/shared/RepoHeader.vue:1-369](), [frontend/src/stores/RepoDetailStore.js:1-246](), [frontend/src/components/shared/RepoSummary.vue:1-261]() + +## Repository Header Component + +The RepoHeader component is a versatile UI element that adapts its display based on the repository type. It presents critical information about a repository such as name, owner, visibility, and status. + +### RepoHeader Structure + +```mermaid +classDiagram + class RepoHeader { + +String avatar + +String name + +String nickname + +String desc + +String path + +String license + +Object tags + +String ownerUrl + +String repoType + +String appStatus + +String spaceResource + +Boolean canWrite + +Number repoId + +Number deployId + +Number collectionsId + +Number totalLikes + +Boolean hasLike + +String resourceName + +copyName() + +clickLike() + +showSpaceLogs() + } + + RepoHeader --> HeaderTags: "uses" + RepoHeader --> AppStatus: "uses" + RepoHeader --> RepoDetailStore: "accesses" +``` + +Sources: [frontend/src/components/shared/RepoHeader.vue:1-180](), [frontend/src/components/shared/RepoHeader.vue:228-368]() + +### Type-Specific Headers + +The RepoHeader adapts to different repository types with conditional rendering: + +1. **Dataset Header** (lines 5-44): Displays dataset icon, name, and metadata +2. **Endpoint Header** (lines 47-74): Shows endpoint status and deployment information +3. **Finetune Header** (lines 77-98): Displays finetune status and resource information +4. **Other Repository Header** (lines 101-157): Handles models, code, spaces with appropriate icons + +Sources: [frontend/src/components/shared/RepoHeader.vue:5-157]() + +### Key Features + +- **Private Repository Indicator**: Shows private badge for private repositories +- **Repository Source Display**: Shows source icons for repositories mirrored from HuggingFace or ModelScope +- **Like Functionality**: Allows users to like/unlike repositories with counter +- **Error Status Indicators**: Displays error tooltips for failed deployments +- **Path Information**: Shows owner and repository path with clickable links +- **Copy Functionality**: Allows copying repository path with a single click + +Sources: [frontend/src/components/shared/RepoHeader.vue:26-43](), [frontend/src/components/shared/RepoHeader.vue:116-143](), [frontend/src/components/shared/RepoHeader.vue:185-196]() + +## RepoDetailStore + +The RepoDetailStore is a centralized state management store built with Pinia that maintains the repository's data and state. It's shared across all repository detail components. + +### Store Structure + +```mermaid +classDiagram + class RepoDetailStore { + +timestamp: Ref~Date~ + +isInitialized: Ref~Boolean~ + +repoType: Ref~String~ + +privateVisibility: Ref~Boolean~ + +baseModel: Ref~String~ + +canManage: Ref~Boolean~ + +canWrite: Ref~Boolean~ + +description: Ref~String~ + +downloads: Ref~Number~ + +hfPath: Ref~String~ + +msPath: Ref~String~ + +name: Ref~String~ + +namespace: Ref~Object~ + +nickname: Ref~String~ + +path: Ref~String~ + +tags: Ref~Array~ + +likes: Ref~Number~ + +userLikes: Ref~Boolean~ + +status: Ref~String~ + +isPrivate(): Boolean + +updateVisibility(input: Boolean) + +updateLikes(input: Number) + +updateUserLikes(input: Boolean) + +initialize(initialData: Object, repositoryType: String) + +clearStore() + } + + note for RepoDetailStore "Contains specific state properties for:\n- Spaces\n- Collections\n- Finetunes\n- Endpoints" +``` + +Sources: [frontend/src/stores/RepoDetailStore.js:6-244]() + +### Repository Types Support + +The store includes specialized fields for various repository types: +- **Common fields**: ID, name, path, description, visibility, etc. +- **Space-specific**: coverImageUrl, endpoint, hardware, sdk, sku, svcName +- **Collections-specific**: avatar, theme, repositories list +- **Finetune-specific**: clusterId, deployId, imageId, replicas, proxyEndpoint +- **Endpoint-specific**: gitBranch, task, replica, instances, engineArgs + +Sources: [frontend/src/stores/RepoDetailStore.js:10-77](), [frontend/src/stores/RepoDetailStore.js:95-158]() + +### State Management Functions + +RepoDetailStore provides several functions to manage repository state: +- `initialize(initialData, repositoryType)`: Sets up the store with data from the backend +- `updateVisibility(input)`: Updates repository visibility +- `updateLikes(input)`: Updates like count +- `updateUserLikes(input)`: Updates user's like status +- `clearStore()`: Resets the store state + +Sources: [frontend/src/stores/RepoDetailStore.js:83-163]() + +## Repository Summary Component + +The RepoSummary component displays the repository's detailed content, including README information, statistics, and related repositories. + +### Structure and Features + +```mermaid +graph TD + RepoSummary[("RepoSummary Component")] + + subgraph "Content Display" + README["README Markdown"] + Downloads["Download Count"] + ParquetViewer["Dataset Preview"] + TestEndpoint["Test Endpoint Interface"] + ApiExample["API Examples"] + end + + subgraph "Related Repositories" + SpaceRelations["Related Spaces"] + CodeRelations["Related Code"] + DatasetRelations["Related Datasets"] + PromptRelations["Related Prompts"] + ModelRelations["Related Models"] + end + + RepoSummary --> README + RepoSummary --> Downloads + RepoSummary --> ParquetViewer + RepoSummary --> TestEndpoint + RepoSummary --> ApiExample + + RepoSummary --> SpaceRelations + RepoSummary --> CodeRelations + RepoSummary --> DatasetRelations + RepoSummary --> PromptRelations + RepoSummary --> ModelRelations +``` + +Sources: [frontend/src/components/shared/RepoSummary.vue:1-219]() + +### Key Components + +1. **Markdown Viewer**: Renders the repository's README file +2. **ParquetViewer**: For dataset repositories, provides a tabular view of the dataset contents +3. **TestEndpoint**: For model repositories with active endpoints, provides an interface for testing the model +4. **Related Repositories**: Shows relationships between the current repository and other repositories + +Sources: [frontend/src/components/shared/RepoSummary.vue:5-14](), [frontend/src/components/shared/RepoSummary.vue:48-71](), [frontend/src/components/datasets/ParquetViewer.vue:1-261]() + +### Data Fetching + +RepoSummary fetches several types of data: +- README content from the repository +- Dataset catalog information (for datasets) +- Repository relations (connected repositories) +- Endpoint information (for models with deployments) + +Sources: [frontend/src/components/shared/RepoSummary.vue:147-196]() + +## Integration with Repository Types + +The Repository Detail Components adapt to different repository types to provide specialized views and functionality. + +### Repository Type Integrations + +| Repository Type | Header Adaptations | Summary Content | Store-specific Fields | +|-----------------|-------------------|----------------|----------------------| +| Model | Model icon, likes display | README, endpoint testing, API examples | enableEvaluation, enableFinetune, enableInference | +| Dataset | Dataset icon, likes display | README, dataset preview (ParquetViewer) | N/A | +| Endpoint | Endpoint icon, status display | Deployment information | endpoint, instances, proxyEndpoint, engineArgs, status | +| Finetune | Finetune icon, status display | Finetune settings | clusterId, deployId, imageId, runtimeFramework | +| Collection | Collection icon, likes display | Collection settings, repositories list | avatar, theme, repositories | + +Sources: [frontend/src/components/shared/RepoHeader.vue:5-157](), [frontend/src/components/finetune/FinetuneDetail.vue:1-128](), [frontend/src/components/endpoints/EndpointDetail.vue:1-63](), [frontend/src/components/collections/CollectionsDetail.vue:1-98]() + +### Repository Type Specific Components + +```mermaid +graph TD + RepoDetail["Repository Detail Page"] + RH["RepoHeader"] + RepoDetail --> RH + + subgraph "Model Repository" + ModelSummary["RepoSummary"] + ModelTabs["RepoTabs"] + ModelSummary --> README["README Content"] + ModelSummary --> TestEndpoint["Endpoint Testing"] + ModelTabs --> Files["Files Tab"] + ModelTabs --> Community["Community Tab"] + ModelTabs --> Settings["Settings Tab"] + end + + subgraph "Endpoint Repository" + EndpointDetail["EndpointDetail"] + EndpointPage["EndpointPage"] + EndpointLogs["EndpointLogs"] + EndpointPlayground["EndpointPlayground"] + EndpointSettings["EndpointSettings"] + EndpointDetail --> EndpointPage + EndpointDetail --> EndpointLogs + EndpointDetail --> EndpointPlayground + EndpointDetail --> EndpointSettings + end + + subgraph "Finetune Repository" + FinetuneTabs["Finetune Tabs"] + FinetuneSettings["FinetuneSettings"] + FinetuneConsole["Finetune Console"] + BillingDetail["Billing Detail"] + FinetuneTabs --> FinetuneConsole + FinetuneTabs --> FinetuneSettings + FinetuneTabs --> BillingDetail + end + + subgraph "Collection Repository" + CollectionTabs["Collection Tabs"] + CollectionSettings["CollectionSettings"] + RepoList["CollectionsRepoList"] + CollectionTabs --> RepoList + CollectionTabs --> CollectionSettings + end + + RepoDetail --> ModelSummary + RepoDetail --> EndpointDetail + RepoDetail --> FinetuneTabs + RepoDetail --> CollectionTabs +``` + +Sources: [frontend/src/components/finetune/FinetuneDetail.vue:30-105](), [frontend/src/components/endpoints/EndpointDetail.vue:17-39](), [frontend/src/components/collections/CollectionsDetail.vue:30-98]() + +## Repository Status Indicators + +The repository components include sophisticated status indicators for deployable repositories (endpoints, finetunes, spaces). + +### Status Display + +Status information is displayed in the header and is kept up-to-date using Server-Sent Events (SSE): + +1. **Status Indicators**: + - Running: Shows active status + - Building/Deploying: Shows loading state + - DeployFailed: Shows failure state with tooltip explanation + - Stopped/Sleeping: Shows inactive state + +2. **Status Messages**: + - Error states show descriptive error messages + - Messages are internationalized with i18n translations + +Sources: [frontend/src/components/finetune/FinetuneDetail.vue:232-280](), [frontend/src/components/endpoints/EndpointDetail.vue:147-192](), [frontend/src/locales/en_js/all.js:112-117](), [frontend/src/locales/zh_js/all.js:112-117]() + +### Error States Handling + +When a deployment fails, the components provide informative error messages: + +``` +deployFailed: 'deploy failed', +Error: 'incomplete model or an incorrect inference engine parameter', +OOMKilled: 'insufficient resources', +CrashLoopBackOff: 'container crashes' +``` + +Sources: [frontend/src/locales/en_js/all.js:112-117](), [frontend/src/components/shared/RepoHeader.vue:67-73]() + +## Internationalization Support + +All repository components are fully internationalized with support for English and Chinese languages. + +### Localized Elements + +Key localized elements include: +- Status messages and tooltips +- Button labels and actions +- Visibility indicators (public/private) +- Settings labels and descriptions + +Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/zh_js/all.js:1-118]() + +## Repository Settings Integration + +For repositories with editable settings, the components integrate with the appropriate settings components. + +### Settings Components + +- **Collection Settings**: Enables editing collection name, description, theme, and visibility +- **Endpoint/Finetune Settings**: Shows deployment configuration and allows updating settings +- **Security Settings**: Controls repository visibility (public/private) + +Sources: [frontend/src/components/collections/CollectionsSettings.vue:1-326]() + +## Conclusion + +The Repository Detail Components system provides a flexible and consistent interface for displaying repository information across different repository types. By using a shared store and adaptable display components, the system can present relevant information for models, datasets, endpoints, finetunes, and collections while maintaining a cohesive user experience. + +The components are designed to handle various repository states, including error conditions, and provide appropriate user feedback. Internationalization support ensures that the interface is accessible to both English and Chinese users. \ No newline at end of file diff --git a/docs/development/4_Repository_Clone_and_Actions.md b/docs/development/4_Repository_Clone_and_Actions.md new file mode 100644 index 000000000..6aad448ab --- /dev/null +++ b/docs/development/4_Repository_Clone_and_Actions.md @@ -0,0 +1,280 @@ +# Repository Clone and Actions + +## Purpose and Scope + +This document explains the Repository Clone and Actions system in CSGHub, which provides users with a unified interface for cloning repositories and performing repository-specific actions like deployment, fine-tuning, and evaluation. It focuses on the UI components, available cloning methods, and action flows for different repository types. + +For information about repository management and detailed views, see [Repository Detail Components](#2.1). For browsing and filtering repositories, see [Repository Browsing and Filtering](#2.3). + +## Component Overview + +The Repository Clone and Actions system consists of a set of user interface components that enable users to interact with repositories in various ways. The system adapts based on repository type (model, dataset, code, space) and the user's permissions. + +```mermaid +graph TD + subgraph "Repository Page" + RT["RepoTabs"] --> RC["RepoClone"] + RT --> TC["TabContainer"] + end + + subgraph "Cloning Methods" + RC --> CD["Clone Dialog"] + CD --> CM1["Command Line Tab"] + CD --> CM2["HTTPS Tab"] + CD --> CM3["SSH Tab"] + CD --> CM4["SDK Tab"] + end + + subgraph "Repository Actions" + RC --> A1["Add to Collections"] + RC --> A2["Sync Repository"] + RC --> A3["Evaluation"] + RC --> A4["Deploy"] + RC --> A5["Fine-tune"] + end + + A4 --> DD["DeployDropdown"] + RC --> RDS["RepoDetailStore"] + RC --> US["UserStore"] +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:1-82](), [frontend/src/components/shared/RepoTabs.vue:1-17]() + +## Repository Clone Button and Dialog + +The clone functionality is accessible via a primary button in the repository interface. When clicked, it opens a dialog with multiple tabs representing different cloning methods. + +```mermaid +sequenceDiagram + participant "User" as U + participant "RepoClone" as RC + participant "CloneDialog" as CD + participant "UserStore" as US + + U->>RC: Click download/clone button + RC->>CD: Open clone dialog + RC->>US: Check authentication status + alt User is logged in + RC->>US: fetchUserToken() + US-->>RC: Return access token + end + U->>CD: Select cloning method + CD-->>U: Display cloning instructions +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:74-82](), [frontend/src/components/shared/RepoClone.vue:84-254]() + +### Cloning Methods + +The system offers four different methods for cloning repositories: + +1. **Command Line** - Uses the CSGHub CLI tool +2. **HTTPS** - Standard Git clone with HTTPS URL +3. **SSH** - Git clone with SSH URL +4. **SDK** - Python SDK for programmatic access + +Each method has specific instructions and commands tailored to the repository type. + +#### Command Line Tab + +Available for models, datasets, and spaces, this tab provides instructions for using the CSGHub CLI tool. + +``` +pip install csghub-sdk +csghub-cli download [repository-path] [-t repository-type] +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:102-142](), [frontend/src/components/shared/RepoClone.vue:383-411]() + +#### HTTPS Tab + +Standard Git clone with HTTPS URL, including instructions for Git LFS and authentication. + +``` +git lfs install +git clone [https-clone-url] +``` + +For authenticated access: +``` +git lfs install +git clone https://[username]:[token]@[repository-url] +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:143-189](), [frontend/src/components/shared/RepoClone.vue:296-360]() + +#### SSH Tab + +Git clone with SSH URL, requiring SSH key setup for authentication. + +``` +git lfs install +git clone [ssh-clone-url] +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:190-223](), [frontend/src/components/shared/RepoClone.vue:303-306]() + +#### SDK Tab + +Available for models and datasets, this tab provides Python code for downloading repository contents using the CSGHub SDK. + +```python +from pycsghub.snapshot_download import snapshot_download +token = '' # token from opencsg.com +endpoint = "https://hub.opencsg.com" +repo_type = "[repository-type]" +repo_id = '[namespace-path]' +cache_dir = '' # cache dir of download data +result = snapshot_download(repo_id, cache_dir=cache_dir, endpoint=endpoint, token=token, repo_type=repo_type) +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:224-253](), [frontend/src/components/shared/RepoClone.vue:370-379]() + +## Repository Actions + +The Repository Clone and Actions system provides various action buttons depending on the repository type and user permissions. + +### Common Actions + +| Action | Description | Availability | +|--------|-------------|--------------| +| Clone/Download | Downloads repository contents | All repositories with HTTP clone URL | +| Add to Collections | Adds repository to user's collections | Models, datasets, codes, spaces | + +Sources: [frontend/src/components/shared/RepoClone.vue:5-9](), [frontend/src/components/shared/RepoClone.vue:74-81]() + +### Model-Specific Actions + +Models have additional actions related to model lifecycle management: + +| Action | Description | Conditions | +|--------|-------------|------------| +| Deploy | Creates an inference endpoint | User logged in, endpoint enabled, HTTP clone URL available | +| Fine-tune | Creates a fine-tuning job | User logged in, fine-tuning enabled, HTTP clone URL available | +| Evaluate | Creates an evaluation job | User logged in, evaluation enabled, HTTP clone URL available | + +```mermaid +flowchart TD + subgraph "Model Repository Actions" + A["User visits model repository"] --> B["View repository details"] + B --> C["Click Deploy button"] + B --> D["Click Fine-tune button"] + B --> E["Click Evaluate button"] + B --> F["Click Download button"] + + C --> C1{"Is user logged in?"} + C1 -->|Yes| C2["Open deployment form"] + C1 -->|No| C3["Redirect to login"] + + D --> D1{"Is user logged in?"} + D1 -->|Yes| D2["Open fine-tuning form"] + D1 -->|No| D3["Redirect to login"] + + E --> E1{"Is user logged in?"} + E1 -->|Yes| E2["Open evaluation form"] + E1 -->|No| E3["Redirect to login"] + + F --> F1["Open clone dialog"] + end +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:22-72](), [frontend/src/components/shared/RepoClone.vue:461-519]() + +### Administrative Actions + +For administrators and super users, additional actions are available: + +| Action | Description | Conditions | +|--------|-------------|------------| +| Sync | Synchronizes repository from remote source | User is admin/super_user, repository source is 'opencsg', sync status is pending/inprogress/failed | + +Sources: [frontend/src/components/shared/RepoClone.vue:11-20](), [frontend/src/components/shared/RepoClone.vue:326-337](), [frontend/src/components/shared/RepoClone.vue:485-503]() + +## Technical Implementation + +### Component Structure + +The Repository Clone and Actions system is implemented primarily through the `RepoClone.vue` component, which is incorporated into the `RepoTabs.vue` component in the repository detail view. + +```mermaid +classDiagram + class RepoClone { + +props: repoType, namespacePath, repo, enableEndpoint, enableFinetune, enableEvaluation, showAddToCollections + +computed: httpCloneUrl, sshCloneUrl, showSyncButton, syncInprogress, actionLimited, isLoggedIn + +methods: handleSyncRepo(), handleButtonClick(), toNewEvaluatePage(), fetchUserToken() + } + + class RepoTabs { + +props: repoDetail, currentBranch, currentPath, defaultTab, tags, actionName, settingsVisibility, canWrite, repoType, appStatus, appEndpoint, sdk, userName + +computed: showAddToCollections + +methods: tabChange() + } + + class TabContainer { + +props: defaultTab, settingsVisibility, repoType, sdk, repo + +computed: summaryLabel, showFiles + +methods: handleTabLeave(), handleTabClick() + } + + RepoTabs --> RepoClone : includes + RepoTabs --> TabContainer : includes +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:258-526](), [frontend/src/components/shared/RepoTabs.vue:251-384](), [frontend/src/components/shared/TabContainer.vue:74-119]() + +### Authentication Flow + +When performing authenticated actions (like deploying a model), the system first checks if the user is logged in: + +1. If the user is logged in, they are directed to the appropriate form +2. If not logged in, they are redirected to the login page + +For cloning with HTTPS authentication, the system fetches the user's access token: + +```mermaid +sequenceDiagram + participant User + participant RepoClone + participant UserStore + participant API + + User->>RepoClone: Click clone button with "Use Token" checked + RepoClone->>UserStore: Check if user is logged in + alt User is logged in + RepoClone->>API: fetchUserToken() + API-->>RepoClone: Return access token + RepoClone-->>User: Display clone command with token + else User is not logged in + RepoClone-->>User: Display standard clone command + end +``` + +Sources: [frontend/src/components/shared/RepoClone.vue:168-175](), [frontend/src/components/shared/RepoClone.vue:468-519](), [frontend/src/components/shared/RepoClone.vue:479-483]() + +### Conditional Rendering + +The Repository Clone and Actions system adapts its interface based on various factors: + +- **Repository Type** - Different actions are available for different repository types +- **User Permissions** - Some actions are only available to logged-in users or administrators +- **Repository State** - Actions may be disabled based on repository status +- **Feature Enablement** - Actions like deployment, fine-tuning, and evaluation can be enabled/disabled + +Sources: [frontend/src/components/shared/RepoClone.vue:22-72](), [frontend/src/components/shared/RepoClone.vue:326-337]() + +## Integration with Other Systems + +The Repository Clone and Actions system integrates with several other systems in CSGHub: + +1. **User Authentication** - Checks login status and retrieves tokens +2. **Collections** - Adds repositories to collections +3. **Deployment** - Initiates model deployment +4. **Fine-tuning** - Starts fine-tuning jobs +5. **Evaluation** - Creates evaluation jobs +6. **Repository Detail** - Obtains repository metadata + +Sources: [frontend/src/components/shared/RepoClone.vue:264-270](), [frontend/src/components/shared/RepoClone.vue:36-40](), [frontend/src/components/shared/RepoClone.vue:65-72]() + +## Summary + +The Repository Clone and Actions system provides a versatile interface for interacting with repositories in CSGHub. It offers multiple cloning methods and repository-specific actions, adapting to the repository type and user permissions. The system is designed to guide users through the appropriate workflows for accessing repository content and performing operations like deployment, fine-tuning, and evaluation. \ No newline at end of file diff --git a/docs/development/5_Repository_Browsing_and_Filtering.md b/docs/development/5_Repository_Browsing_and_Filtering.md new file mode 100644 index 000000000..25a369782 --- /dev/null +++ b/docs/development/5_Repository_Browsing_and_Filtering.md @@ -0,0 +1,346 @@ +# Repository Browsing and Filtering + +This document covers the repository browsing and filtering functionality in CSGHub, explaining how users can discover, search, and filter repositories (models, datasets, code, and spaces) across the platform. For information about detailed repository views, see [Repository Detail Components](#2.1), and for clone functionality, see [Repository Clone and Actions](#2.2). + +## Overview + +The repository browsing system enables users to efficiently navigate through CSGHub's repository collection with robust filtering, sorting, and search capabilities. The system is built with reusable components that adapt to different repository types while maintaining consistent user experience. + +```mermaid +flowchart TD + User(["User"]) -- "Browses/searches" --> RepoCards["RepoCards.vue"] + RepoCards -- "Displays" --> RepositoryList["Repository List"] + User -- "Applies filters" --> TagSidebar["TagSidebar.vue"] + TagSidebar -- "Updates filters" --> RepoCards + RepoCards -- "Fetches data" --> API["Repository API"] + User -- "Changes page" --> Pagination["CsgPagination.vue"] + Pagination -- "Updates page" --> RepoCards + RepoCards -- "Updates search params" --> UrlParams["URL Parameters"] +``` + +Sources: [frontend/src/components/shared/RepoCards.vue:1-332](). [frontend/src/components/tags/TagSidebar.vue:1-188]() + +## Key Components + +### Repository Cards Component + +The `RepoCards.vue` component serves as the central hub for repository browsing functionality. It handles: + +1. Displaying repository cards in a responsive grid layout +2. Managing filtering and sorting controls +3. Coordinating with the tag sidebar for category-based filtering +4. Implementing pagination of results +5. Processing search queries + +```mermaid +classDiagram + class RepoCards { + +String repoType + +Array reposData + +Number totalRepos + +String nameFilterInput + +String sortSelection + +String filterSelection + +String sourceSelection + +Number currentPage + +Object activeTags + +Boolean loading + +filterChange() + +reloadRepos(currentPage) + +loadRepos(url) + +resetTags(tags) + } + + class TagSidebar { + +String repoType + +String selectedTag + +String selectedTagType + +Array tagCategories + +Object tagsForCategory + +String activeNavItem + +Object activeTags + +changeActiveItem() + +setActiveTag() + +fetchTags() + +fetchTagCategories() + +emitTag() + } + + class CsgPagination { + +Number perPage + +Number currentPage + +Number total + +handleCurrentChange() + } + + RepoCards --> TagSidebar : uses + RepoCards --> CsgPagination : uses +``` + +Sources: [frontend/src/components/shared/RepoCards.vue:1-332](). [frontend/src/components/tags/TagSidebar.vue:1-188]() + +### Repository Item Rendering + +The system adapts the display based on repository type: + +- For models, datasets, and code repositories, it uses the standard `RepoItem` component +- For spaces, it uses the specialized `ApplicationSpaceItem` component +- Empty states are handled with specific components like `EmptyModels` + +Different layouts are applied based on repository type and screen size: + +``` +models, datasets, code: grid-cols-2 xl:grid-cols-1 +spaces: grid-cols-3 xl:grid-cols-2 md:grid-cols-1 +``` + +Sources: [frontend/src/components/shared/RepoCards.vue:105-123](). [frontend/src/components/models/EmptyModels.vue:1-15]() + +## Filtering Capabilities + +### Search and Sort + +The system provides multiple filtering dimensions: + +1. **Text Search**: Allows free-text search across repository names and descriptions +2. **Sorting Options**: + - Trending (default) + - Recently updated + - Most downloaded + - Most favorited + +```javascript +const sortOptions = [ + { value: 'trending', label: t('all.trending') }, + { value: 'recently_update', label: t('all.recentlyUpdate') }, + { value: 'most_download', label: t('all.mostDownload') }, + { value: 'most_favorite', label: t('all.mostFavorite') } +] +``` + +3. **Source Filtering** (on-premise installations): + - All + - OpenCSG + - Local + +Sources: [frontend/src/components/shared/RepoCards.vue:46-83](). [frontend/src/components/shared/RepoCards.vue:174-238]() + +### Tag-Based Filtering + +The `TagSidebar` component enables powerful category-based filtering: + +1. **Tag Categories**: Organized into categories (task, framework, language, license, etc.) +2. **Multi-select**: Users can select multiple tags within categories +3. **Tag Search**: Users can filter available tags with a search input + +For framework tags, the system includes specialized visual components for popular frameworks: + +- PyTorch +- TensorFlow +- Safetensors +- ONNX +- JAX +- PaddlePaddle +- GGUF +- Joblib + +```mermaid +flowchart TD + subgraph "Tag Filter System" + TagSidebar["TagSidebar.vue"] + TagCategory["TagCategory.vue"] + TagList["TagList.vue"] + FrameworkTags["Framework Tag Components"] + TagItem["TagItem.vue"] + + TagSidebar --> TagCategory + TagSidebar --> TagList + TagList --> FrameworkTags + TagList --> TagItem + end + + FrameworkTags --> PyTorch["PyTorch.vue"] + FrameworkTags --> TensorFlow["TensorFlow.vue"] + FrameworkTags --> Safetensors["Safetensors.vue"] + FrameworkTags --> ONNX["ONNX.vue"] + FrameworkTags --> JAX["JAX.vue"] + FrameworkTags --> Others["Other Framework Tags"] + + TagSidebar -- "Selected Tags" --> RepoCards["RepoCards.vue"] +``` + +Sources: [frontend/src/components/tags/TagSidebar.vue:1-188](). [frontend/src/components/tags/TagList.vue:1-159](). [frontend/src/components/tags/frameworks/PyTorch.vue:1-52](). [frontend/src/components/tags/frameworks/TensorFlow.vue:1-26]() + +### Special Filters by Repository Type + +Different repository types have specialized filtering options: + +1. **Models**: + - Runtime framework (inference, finetune, evaluation) + - Serverless availability + +2. **Spaces**: + - SDK type (Gradio, Streamlit, Nginx, Docker, MCP Server) + +Sources: [frontend/src/components/shared/RepoCards.vue:71-83](). [frontend/src/components/shared/RepoCards.vue:217-238]() + +## Implementation Details + +### API Integration + +The repository browsing system interacts with the backend API through the following pattern: + +1. Constructing a URL with query parameters based on current filter state +2. Fetching data using the `useFetchApi` utility +3. Updating the UI with the fetched results + +```javascript +// URL construction pattern +let url = `/${props.repoType}s` +url = url + `?page=${currentPage.value}` +url = url + `&per=${perPage.value}` +url = url + `&search=${nameFilterInput.value}` +url = url + `&sort=${sortSelection.value}` +// Additional parameters based on filters +``` + +Sources: [frontend/src/components/shared/RepoCards.vue:272-308](). [frontend/src/components/shared/RepoCards.vue:311-325]() + +### Pagination + +The system implements pagination with the following features: + +1. Dynamic per-page count based on repository type: + - 9 items per page for spaces + - 16 items per page for other repository types + +2. Page navigation through the `CsgPagination` component + +The pagination component tracks: +- Current page +- Items per page +- Total item count +- Emits events for page changes + +Sources: [frontend/src/components/shared/RepoCards.vue:255-261](). [frontend/src/components/shared/RepoCards.vue:124-128]() + +### URL Parameter Integration + +The system supports URL parameters for deep linking and sharing filtered views: + +1. When a page loads, it reads URL parameters to set initial filter state: + - `tag`: Selected tag value + - `tag_type`: Type of tag (framework, task, etc.) + +2. Active filters are synchronized with URL parameters for bookmarking and sharing + +```javascript +const getQueryParams = () => { + const { searchParams } = new URL(window.location.href) + return { + tag: searchParams.get('tag') ?? '', + tagType: searchParams.get('tag_type') ?? '' + } +} +``` + +Sources: [frontend/src/components/shared/RepoCards.vue:148-156](). [frontend/src/components/tags/TagSidebar.vue:94-115]() + +## Tag System Architecture + +Tags are organized into a hierarchical structure that enables powerful filtering: + +1. **Categories**: Top-level organization (task, framework, language, license) +2. **Groups**: Sub-categories within task tags (e.g., computer_vision, natural_language_processing) +3. **Tags**: Individual selectable options + +The system fetches tags based on scope (model, dataset, code, space) and category. + +```mermaid +flowchart TD + subgraph "Tag Hierarchy" + Category["Tag Category"] + Group["Tag Group (for tasks)"] + Tag["Individual Tag"] + + Category --> Group + Group --> Tag + end + + subgraph "Example Structure" + Task["Task Category"] + CV["Computer Vision Group"] + NLP["NLP Group"] + ObjectDetection["Object Detection Tag"] + ImageClassification["Image Classification Tag"] + TextGeneration["Text Generation Tag"] + + Task --> CV + Task --> NLP + CV --> ObjectDetection + CV --> ImageClassification + NLP --> TextGeneration + end +``` + +Sources: [frontend/src/components/tags/TagSidebar.vue:117-151](). [frontend/src/components/tags/TagList.vue:1-159]() + +## Localization Support + +The repository browsing system fully supports internationalization: + +1. Text labels use the Vue i18n system with the `$t` function +2. Localized strings are defined in language-specific files: + - English: 'models.js', etc. + - Chinese: 'zh_js/models.js', etc. + +This enables a consistent experience across languages while maintaining the same filtering capabilities. + +Sources: [frontend/src/locales/en_js/models.js:1-75](). [frontend/src/locales/zh_js/models.js:1-72]() + +## Collections Integration + +In addition to browsing individual repositories, the system also supports browsing collections: + +- Collections group related repositories together +- Similar filtering patterns are used (search, sort, pagination) +- Collections have their own specialized card display component + +```javascript +// Collection filtering pattern +const params = new URLSearchParams() +params.append('per', perPage.value) +params.append('page', currentPage.value) +params.append('search', nameFilterInput.value) +params.append('sort', sortSelection.value) +const url = `/collections?${params.toString()}` +``` + +Sources: [frontend/src/components/collections/CollectionIndex.vue:1-166]() + +## Header Tags Display + +When browsing repositories with active tag filters, the `HeaderTags` component provides a visual representation of the currently applied tags, organized by category: + +- Task tags +- Framework tags (with specialized icons) +- Language tags +- Industry tags +- License tags + +This component also enables quick removal of tag filters. + +Sources: [frontend/src/components/shared/HeaderTags.vue:1-244]() + +## Summary + +The Repository Browsing and Filtering system in CSGHub provides a comprehensive and flexible way for users to discover and filter repositories. Key features include: + +1. Responsive grid layout adapting to different repository types +2. Powerful filtering through tags, search, and sorting +3. Type-specific filters for specialized repository types +4. Pagination for navigating large result sets +5. URL parameter integration for sharing filtered views +6. Internationalization support for multiple languages + +The component-based architecture allows for reuse across different repository types while maintaining consistent user experience. \ No newline at end of file diff --git a/docs/development/6_Model_Deployment_and_Fine-tuning.md b/docs/development/6_Model_Deployment_and_Fine-tuning.md new file mode 100644 index 000000000..7a6f240a9 --- /dev/null +++ b/docs/development/6_Model_Deployment_and_Fine-tuning.md @@ -0,0 +1,525 @@ +# Model Deployment and Fine-tuning + +This document covers the systems and processes for deploying models as inference endpoints and fine-tuning models in CSGHub. It explains how to create, configure, and manage both inference endpoints and fine-tuning jobs, as well as how to evaluate models. + +For repository management details, see [Repository Management System](#2). + +## Overview + +CSGHub provides three main AI model operations: + +1. **Model Deployment** - Deploy models as inference endpoints for real-time prediction +2. **Model Fine-tuning** - Customize pre-trained models with your own data +3. **Model Evaluation** - Measure model performance with standardized metrics + +These operations share common infrastructure components but serve different purposes in the ML lifecycle. + +```mermaid +flowchart TD + User(["User"]) + + subgraph "Model Operations" + Deploy["Model Deployment"] + Finetune["Model Fine-tuning"] + Evaluate["Model Evaluation"] + end + + subgraph "Shared Components" + Resources["Resource Management"] + Frameworks["Runtime Frameworks"] + EngineArgs["Engine Arguments"] + end + + subgraph "Infrastructure" + Clusters["Cluster Management"] + Hardware["Hardware Provisioning"] + Scaling["Replica Scaling"] + end + + User --> Deploy + User --> Finetune + User --> Evaluate + + Deploy --> Resources + Finetune --> Resources + Evaluate --> Resources + + Deploy --> Frameworks + Finetune --> Frameworks + Evaluate --> Frameworks + + Deploy --> EngineArgs + Finetune --> EngineArgs + + Resources --> Clusters + Resources --> Hardware + Deploy --> Scaling +``` + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:1-611]() +- [frontend/src/components/finetune/NewFinetune.vue:1-416]() +- [frontend/src/components/evaluations/NewEvaluation.vue:1-553]() +- [frontend/src/components/endpoints/EngineArgs.vue:1-128]() + +## Inference Endpoint System + +Inference endpoints provide a deployable API for using models in production. The endpoint system handles deployment configuration, resource allocation, and replica scaling. + +### Creating an Endpoint + +The endpoint creation flow allows users to deploy a model with specific hardware resources and runtime parameters: + +```mermaid +sequenceDiagram + participant User + participant NewEndpoint as "NewEndpoint Component" + participant API as "Backend API" + participant ResourceSystem as "Resource System" + participant ClusterSystem as "Cluster System" + + User->>NewEndpoint: Enter endpoint name and model ID + NewEndpoint->>API: fetchModels(query) + API-->>NewEndpoint: Return matching models + + User->>NewEndpoint: Select cluster region + NewEndpoint->>API: fetchClusters() + API-->>NewEndpoint: Return available clusters + NewEndpoint->>API: fetchResources(clusterId) + API-->>NewEndpoint: Return available hardware resources + + User->>NewEndpoint: Select hardware resource + User->>NewEndpoint: Configure min/max replicas + + NewEndpoint->>API: fetchRuntimeFramework(modelId) + API-->>NewEndpoint: Return compatible frameworks + User->>NewEndpoint: Select runtime framework + + opt Framework has custom parameters + NewEndpoint->>NewEndpoint: Display EngineArgs component + User->>NewEndpoint: Configure framework parameters + end + + opt Model has quantization options + NewEndpoint->>API: fetchQuantizations(modelId) + API-->>NewEndpoint: Return available quantizations + User->>NewEndpoint: Select quantization option + end + + User->>NewEndpoint: Set visibility (public/private) + User->>NewEndpoint: Click Create + + NewEndpoint->>API: createEndpoint(params) + API-->>NewEndpoint: Return deployment status + NewEndpoint-->>User: Redirect to endpoint detail page +``` + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:1-611]() +- [frontend/src/components/endpoints/EngineArgs.vue:1-128]() + +The form allows configuration of: + +| Parameter | Description | Default | +|-----------|-------------|---------| +| Endpoint name | Unique identifier for the endpoint | - | +| Model ID | The model to deploy | - | +| Min Replicas | Minimum number of running instances | 1 | +| Max Replicas | Maximum number of instances for scaling | 1 | +| Cluster | Region for deployment | First available | +| Cloud Resource | Hardware resource type | First available | +| Runtime Framework | Inference framework | Model-dependent | +| Engine Arguments | Framework-specific parameters | Framework defaults | +| Quantization | Optional model quantization | None | +| Visibility | Public or private access | Public | + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:16-265]() +- [frontend/src/locales/en_js/endpoints.js:1-90]() + +### Managing Endpoints + +The endpoint settings interface provides controls for managing deployed endpoints: + +```mermaid +graph TD + subgraph "EndpointSettings Component" + StopAction["Stop Endpoint"] + RestartAction["Restart Endpoint"] + ResourceConfig["Change Resource Configuration"] + FrameworkConfig["Change Runtime Framework"] + EngineConfig["Update Engine Arguments"] + ReplicaConfig["Adjust Min/Max Replicas"] + VisibilityConfig["Change Visibility"] + DeleteAction["Delete Endpoint"] + end + + subgraph "Endpoint API Actions" + StopAPI["stopEndpoint()"] + RestartAPI["restartEndpoint()"] + UpdateAPI["updateEndpoint(payload)"] + DeleteAPI["deleteEndpoint()"] + end + + StopAction --> StopAPI + RestartAction --> RestartAPI + + ResourceConfig --> UpdateAPI + FrameworkConfig --> UpdateAPI + EngineConfig --> UpdateAPI + ReplicaConfig --> UpdateAPI + VisibilityConfig --> UpdateAPI + + DeleteAction --> DeleteAPI + + UpdateAPI --> RefreshEndpoint["Refresh Endpoint Data"] +``` + +Sources: +- [frontend/src/components/endpoints/EndpointSettings.vue:1-573]() + +Key management operations include: + +1. **State Management**: Stop, start, and restart the endpoint +2. **Resource Management**: Change the hardware resources (only when stopped) +3. **Framework Configuration**: Change the runtime framework and engine arguments +4. **Scaling Configuration**: Adjust the minimum and maximum replicas +5. **Visibility Control**: Toggle between public and private access +6. **Deletion**: Permanently remove the endpoint + +Sources: +- [frontend/src/components/endpoints/EndpointSettings.vue:24-276]() + +### Endpoint Lifecycle and States + +The endpoint goes through several states during its lifecycle: + +| State | Description | Actions Available | +|-------|-------------|-------------------| +| Building | Initial build process | None | +| Deploying | Deployment in progress | None | +| Startup | Starting the service | None | +| Running | Active and serving requests | Stop, monitoring | +| Stopped | Endpoint is paused | Start, configure, delete | +| Sleeping | Low-activity state | Start | +| BuildingFailed | Build process failed | Restart, delete | +| DeployFailed | Deployment failed | Restart, delete | +| RuntimeError | Error during operation | Restart, delete | +| NoAppFile | Missing application file | Delete | + +Sources: +- [frontend/src/components/endpoints/EndpointSettings.vue:334-354]() +- [frontend/src/components/endpoints/EndpointPage.vue:1-142]() + +## Fine-tuning System + +The fine-tuning system allows users to customize pre-trained models with their own data. + +### Creating a Fine-tuning Job + +```mermaid +graph TD + subgraph "NewFinetune Component" + ModelSelection["Select Base Model"] + NameConfig["Configure Job Name"] + ClusterSelection["Select Region/Cluster"] + ResourceSelection["Select Hardware Resource"] + FrameworkSelection["Select Runtime Framework"] + end + + subgraph "API Interactions" + FetchModels["fetchModels()"] + FetchClusters["fetchClusters()"] + FetchResources["fetchResources(clusterId)"] + FetchFrameworks["fetchRuntimeFramework(modelId)"] + CreateFinetune["submitFinetuneForm()"] + end + + ModelSelection --> FetchModels + ModelSelection --> FetchFrameworks + + ClusterSelection --> FetchClusters + ClusterSelection --> FetchResources + + ResourceSelection --> FilterFrameworks["Filter compatible frameworks"] + + ModelSelection & NameConfig & ClusterSelection & ResourceSelection & FrameworkSelection --> CreateFinetune + + CreateFinetune --> Redirect["Redirect to finetune detail page"] +``` + +Sources: +- [frontend/src/components/finetune/NewFinetune.vue:1-416]() + +The fine-tuning form allows configuration of: + +| Parameter | Description | Default | +|-----------|-------------|---------| +| Job Name | Name for the fine-tuning job | - | +| Model ID | Base model to fine-tune | - | +| Cluster | Region for fine-tuning | First available | +| Cloud Resource | Hardware resource (GPU required) | First available | +| Runtime Framework | Fine-tuning framework | Model-dependent | + +Sources: +- [frontend/src/components/finetune/NewFinetune.vue:16-164]() + +### Managing Fine-tuning Jobs + +The fine-tuning settings interface provides controls similar to endpoint management: + +1. **State Management**: Stop and restart the fine-tuning job +2. **View Configuration**: Examine the resource allocation and framework settings +3. **Deletion**: Permanently remove the fine-tuning job + +Fine-tuning jobs share the same lifecycle states as endpoints (Building, Running, Stopped, etc.). + +Sources: +- [frontend/src/components/finetune/FinetuneSettings.vue:1-387]() + +## Model Evaluation System + +The evaluation system allows users to measure model performance against standard benchmarks. + +### Creating an Evaluation + +```mermaid +graph TD + subgraph "NewEvaluation Component" + NameConfig["Configure Evaluation Name"] + ModelSelection["Select Model to Evaluate"] + DescConfig["Add Description"] + DatasetSelection["Select Evaluation Datasets"] + ResourceTypeSelection["Choose Resource Type"] + + subgraph "For Dedicated Resources" + ClusterSelection["Select Region/Cluster"] + ResourceSelection["Select Hardware Resource"] + end + + FrameworkSelection["Select Evaluation Framework"] + end + + subgraph "API Interactions" + FetchModels["fetchModels()"] + FetchDatasets["fetchDatasetOptions()"] + FetchClusters["fetchClusters()"] + FetchResources["fetchResources()"] + FetchFrameworks["fetchRuntimeFramework()"] + SubmitEvaluation["handleSubmit()"] + end + + ModelSelection --> FetchModels + ModelSelection --> FetchFrameworks + FrameworkSelection --> FetchDatasets + + ResourceTypeSelection --> ResourceTypeDecision{"Dedicated or Shared?"} + ResourceTypeDecision -->|"Dedicated"| ClusterSelection + ClusterSelection --> FetchClusters + ClusterSelection --> FetchResources + + NameConfig & ModelSelection & DescConfig & DatasetSelection & ResourceTypeSelection & FrameworkSelection --> SubmitEvaluation + ResourceTypeDecision -->|"Dedicated"| ResourceSelection --> SubmitEvaluation + + SubmitEvaluation --> Redirect["Redirect to resource console"] +``` + +Sources: +- [frontend/src/components/evaluations/NewEvaluation.vue:1-553]() + +The evaluation form allows configuration of: + +| Parameter | Description | Required | +|-----------|-------------|----------| +| Evaluation Name | Name for the evaluation task | Yes | +| Model ID | Model to evaluate | Yes | +| Description | Purpose and notes for evaluation | No | +| Datasets | Evaluation datasets (filtered by framework) | Yes | +| Resource Type | Shared (pooled) or Dedicated resources | Yes | +| Cluster | Region (for dedicated resources) | Conditional | +| Cloud Resource | Hardware (for dedicated resources) | Conditional | +| Evaluation Framework | Framework for running evaluations | Yes | + +Sources: +- [frontend/src/components/evaluations/NewEvaluation.vue:16-244]() + +## Shared Components and Infrastructure + +### Engine Arguments System + +The Engine Arguments component allows configuration of framework-specific parameters for both endpoints and fine-tuning jobs: + +```mermaid +graph TD + subgraph "EngineArgs Component" + ArgsCollapse["Expandable Parameter Section"] + ArgsInputs["Parameter Input Fields"] + end + + subgraph "Input Types" + TextInput["Text Input Fields"] + SelectInput["Select Dropdowns"] + SwitchInput["Toggle Switches"] + end + + subgraph "Parameters Management" + ArgValues["Current Parameter Values"] + ChangedArgs["Track Changed Parameters"] + EmitChanges["Emit Updates to Parent"] + end + + ArgsCollapse --> ArgsInputs + ArgsInputs --> TextInput & SelectInput & SwitchInput + + TextInput & SelectInput & SwitchInput --> HandleChange["handleChange()"] + HandleChange --> ChangedArgs + ChangedArgs --> EmitChanges + + useEngineArgs["getInputTypeForEngineArg()"] --> SelectInput & SwitchInput & TextInput +``` + +Sources: +- [frontend/src/components/endpoints/EngineArgs.vue:1-128]() +- [frontend/src/packs/useEngineArgs.js:1-55]() + +The component intelligently selects input types based on parameter names and values: + +| Input Type | Used For | Example Parameters | +|------------|----------|-------------------| +| Select Dropdown | Parameters with predefined options | `block-size`, `dtype`, `load-format` | +| Toggle Switch | Boolean parameters | `enable-prefix-caching`, `enforce-eager` | +| Text Input | All other parameters | Custom values, numeric settings | + +Sources: +- [frontend/src/packs/useEngineArgs.js:1-55]() + +### Resource and Cluster Management + +All three systems (deployment, fine-tuning, evaluation) share common resource selection and cluster management: + +```mermaid +graph TD + subgraph "Resource Selection Flow" + SelectCluster["Select Cluster/Region"] + FetchResources["fetchResourcesInCategory()"] + FilterResources["Filter By Operation Type"] + DisplayResources["Display Available Resources"] + SelectResource["Select Resource"] + FilterFrameworks["Filter Compatible Frameworks"] + end + + SelectCluster --> FetchResources + FetchResources --> FilterResources + FilterResources --> DisplayResources + DisplayResources --> SelectResource + SelectResource --> FilterFrameworks + + subgraph "Resource Categories" + CPU["CPU Resources"] + GPU["GPU Resources"] + Premium["Premium Resources"] + end + + subgraph "Operation Types" + DeployType["Deployment (Type 1)"] + FinetuneType["Fine-tuning (Type 2)"] + EvalType["Evaluation (Type 4)"] + end + + FilterResources --> CPU & GPU & Premium + DeployType & FinetuneType & EvalType --> FilterResources +``` + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:397-408]() +- [frontend/src/components/finetune/NewFinetune.vue:257-269]() +- [frontend/src/components/evaluations/NewEvaluation.vue:380-390]() + +Key aspects of resource management: + +1. **Clusters**: Geographic regions where resources are deployed +2. **Resource Types**: Hardware configurations (CPU/GPU/specialized accelerators) +3. **Framework Compatibility**: Filtering frameworks based on selected resources +4. **Operation Types**: Different operations (deployment, fine-tuning, evaluation) filter for appropriate resources + +Sources: +- [frontend/src/components/shared/deploy_instance/fetchResourceInCategory.js]() (referenced but not shown in provided files) + +## Integration with Repository System + +The deployment and fine-tuning systems integrate with the repository management system: + +1. **Model Selection**: Deployed endpoints and fine-tuning jobs use models from the repository +2. **Dataset Selection**: Evaluations and fine-tuning use datasets from the repository +3. **Visibility Control**: Aligns with repository visibility settings (public/private) + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:246-252]() +- [frontend/src/components/endpoints/EndpointSettings.vue:185-224]() + +## User Interface and Localization + +The system provides a fully localized interface with support for both English and Chinese: + +```mermaid +graph TD + subgraph "UI Components" + NewEndpoint["NewEndpoint.vue"] + EndpointSettings["EndpointSettings.vue"] + NewFinetune["NewFinetune.vue"] + NewEvaluation["NewEvaluation.vue"] + end + + subgraph "Localization System" + EN["English Locales"] + ZH["Chinese Locales"] + I18n["Vue i18n"] + end + + ZH --> I18n + EN --> I18n + + I18n --> NewEndpoint + I18n --> EndpointSettings + I18n --> NewFinetune + I18n --> NewEvaluation +``` + +Sources: +- [frontend/src/locales/en_js/endpoints.js:1-90]() +- [frontend/src/locales/zh_js/endpoints.js:1-90]() + +## API Integration + +All model operations interact with backend API endpoints: + +| Operation | API Endpoint | HTTP Method | +|-----------|--------------|-------------| +| Create Endpoint | `/models/{modelId}/run` | POST | +| Update Endpoint | `/models/{modelId}/run/{endpointId}` | PUT | +| Stop Endpoint | `/models/{modelId}/run/{endpointId}/stop` | PUT | +| Start Endpoint | `/models/{modelId}/run/{endpointId}/start` | PUT | +| Delete Endpoint | `/models/{modelId}/run/{endpointId}` | DELETE | +| Create Fine-tune | `/models/{modelId}/finetune` | POST | +| Update Fine-tune | `/models/{modelId}/finetune/{finetuneId}` | PUT | +| Stop Fine-tune | `/models/{modelId}/finetune/{finetuneId}/stop` | PUT | +| Start Fine-tune | `/models/{modelId}/finetune/{finetuneId}/start` | PUT | +| Delete Fine-tune | `/models/{modelId}/finetune/{finetuneId}` | DELETE | +| Create Evaluation | `/evaluations` | POST | + +Sources: +- [frontend/src/components/endpoints/NewEndpoint.vue:550-553]() +- [frontend/src/components/endpoints/EndpointSettings.vue:398-400]() +- [frontend/src/components/finetune/NewFinetune.vue:353-359]() +- [frontend/src/components/finetune/FinetuneSettings.vue:285-288]() +- [frontend/src/components/evaluations/NewEvaluation.vue:472-473]() + +## Summary + +The Model Deployment and Fine-tuning systems in CSGHub provide comprehensive capabilities for operationalizing AI models through: + +1. **Inference Endpoints** for real-time predictions +2. **Fine-tuning Jobs** for customizing models with specialized data +3. **Evaluation Tasks** for measuring model performance + +These systems share common infrastructure components including resource management, runtime frameworks, and engine argument configuration, while providing specialized interfaces for each operation type. \ No newline at end of file diff --git a/docs/development/7_Creating_and_Managing_Endpoints.md b/docs/development/7_Creating_and_Managing_Endpoints.md new file mode 100644 index 000000000..c513516b7 --- /dev/null +++ b/docs/development/7_Creating_and_Managing_Endpoints.md @@ -0,0 +1,350 @@ +# Creating and Managing Endpoints + +This page documents the process of creating, configuring, managing, and deleting inference endpoints in CSGHub. Inference endpoints allow you to deploy models for real-time inference, making them accessible via API calls. For information about model fine-tuning, see [Model Fine-tuning](#3.2). + +## 1. Overview of Endpoints + +Inference endpoints in CSGHub are dedicated instances that host models for inference. They provide a REST API for sending inference requests to your deployed models. Endpoints can be configured with different resources, frameworks, and settings to optimize for performance, cost, and availability. + +```mermaid +flowchart TD + User[["User"]] --> CreateEndpoint["Create Endpoint"] + User --> ManageEndpoint["Manage Endpoint"] + User --> UseEndpoint["Use Endpoint"] + + subgraph "Endpoint Lifecycle" + CreateEndpoint --> Building["Building"] + Building --> Deploying["Deploying"] + Deploying --> Running["Running"] + Running --> Stopped["Stopped"] + Stopped --> Running + + Building --> |"Failure"| BuildingFailed["BuildingFailed"] + Deploying --> |"Failure"| DeployFailed["DeployFailed"] + Running --> |"Error"| RuntimeError["RuntimeError"] + end + + subgraph "Endpoint Usage" + UseEndpoint --> API["REST API Calls"] + UseEndpoint --> Playground["Endpoint Playground"] + end + + classDef states fill:#f9f9f9,stroke:#333,stroke-width:1px; + class Building,Deploying,Running,Stopped,BuildingFailed,DeployFailed,RuntimeError states; +``` + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:334-346](), [frontend/src/components/endpoints/EndpointPage.vue:64-73]() + +## 2. Creating an Endpoint + +Creating an endpoint involves specifying the model to deploy, configuring the resources, and setting various parameters to control how the model is deployed and accessed. + +### 2.1 Endpoint Creation Flow + +```mermaid +sequenceDiagram + participant User + participant UI as "NewEndpoint Component" + participant API as "Backend API" + participant Cluster as "Endpoint Cluster" + + User->>UI: Enter endpoint details + UI->>API: Fetch available clusters + API-->>UI: Return cluster list + UI->>API: Fetch available resources + API-->>UI: Return resource list + UI->>API: Fetch runtime frameworks + API-->>UI: Return compatible frameworks + + User->>UI: Select model, cluster, resources, framework + User->>UI: Configure replicas, visibility + User->>UI: Submit endpoint creation + + UI->>API: POST /models/{modelId}/run + API->>Cluster: Deploy model + Cluster-->>API: Deployment initialized + API-->>UI: Return endpoint ID and status + UI->>User: Redirect to endpoint detail page +``` + +Sources: [frontend/src/components/endpoints/NewEndpoint.vue:526-566]() + +### 2.2 Required Parameters + +When creating an endpoint, you need to specify the following parameters: + +| Parameter | Description | Required | +|-----------|-------------|----------| +| Endpoint Name | A unique name for your endpoint (2-64 characters) | Yes | +| Model ID | The ID of the model to deploy (in format `owner/model`) | Yes | +| Min Replica | Minimum number of replicas (0-5) | Yes | +| Max Replica | Maximum number of replicas (1-5) | Yes | +| Cluster | Region where the endpoint will be deployed | Yes | +| Cloud Resource | Hardware resources for the endpoint | Yes | +| Runtime Framework | Framework used to run the model | Yes | +| Visibility | Public or private access | Yes | +| Quantization | Optional model quantization for optimization | No | + +Sources: [frontend/src/components/endpoints/NewEndpoint.vue:25-243]() + +### 2.3 Endpoint Creation Form + +To create a new endpoint, follow these steps: + +1. Navigate to the endpoint creation page. +2. Enter a unique name for your endpoint, following naming rules: + - 2-64 characters long + - Start with a letter + - End with a letter or number + - Can only contain letters, numbers, hyphens, underscores, and periods + - Special characters cannot appear together + +3. Enter the model ID in the format `owner/model`. +4. Select minimum and maximum replica counts. +5. Choose a cluster region for deployment. +6. Select a cloud resource configuration based on your performance needs. +7. Choose a compatible runtime framework. +8. Configure optional engine arguments if needed. +9. Select a quantization option if available. +10. Choose visibility (public or private). +11. Click "Create" to deploy the endpoint. + +Sources: [frontend/src/components/endpoints/NewEndpoint.vue:1-266](), [frontend/src/components/endpoints/NewEndpoint.vue:306-345]() + +### 2.4 Engine Arguments + +Depending on the selected runtime framework, you can configure various engine arguments to optimize performance. These arguments are framework-specific and can include: + +- Block size +- Data type +- Scheduling policy +- Memory utilization +- Caching options + +```mermaid +flowchart TD + NewEndpoint["NewEndpoint.vue"] --> EngineArgs["EngineArgs.vue"] + EngineArgs --> useEngineArgs["useEngineArgs.js"] + + subgraph "Engine Arguments Flow" + direction TB + RuntimeFramework["Runtime Framework Selection"] --> FrameworkEngineArgs["Load Framework Engine Args"] + FrameworkEngineArgs --> ArgTypes["Determine Argument Types"] + ArgTypes --> |"Select"| SelectUI["Dropdown UI"] + ArgTypes --> |"Switch"| SwitchUI["Toggle Switch UI"] + ArgTypes --> |"Text"| TextUI["Text Input UI"] + SelectUI & SwitchUI & TextUI --> ArgValues["Capture Argument Values"] + ArgValues --> CreatePayload["Include in Endpoint Creation Payload"] + end +``` + +Sources: [frontend/src/components/endpoints/EngineArgs.vue:1-127](), [frontend/src/packs/useEngineArgs.js:1-54]() + +## 3. Managing Endpoints + +After creating an endpoint, you can manage its state, update its configuration, and monitor its status. + +### 3.1 Endpoint States + +Endpoints can be in one of several states: + +| State | Description | +|-------|-------------| +| Building | The endpoint is being built | +| Deploying | The endpoint is being deployed | +| Startup | The endpoint is starting up | +| Running | The endpoint is running and available for inference | +| Stopped | The endpoint is stopped | +| Sleeping | The endpoint is in sleep mode | +| BuildingFailed | The endpoint failed during building | +| DeployFailed | The endpoint failed during deployment | +| RuntimeError | The endpoint encountered a runtime error | + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:334-345]() + +### 3.2 Controlling Endpoint State + +You can control the state of your endpoint using the following actions: + +1. **Stop Endpoint**: Pauses the endpoint and stops all running instances. +2. **Restart Endpoint**: Restarts a stopped endpoint or restarts a running endpoint. + +These actions can be performed from the endpoint settings page. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:4-41]() + +### 3.3 Updating Endpoint Configuration + +You can update various aspects of an endpoint's configuration after it has been created. Note that some changes require the endpoint to be in the "Stopped" state. + +```mermaid +flowchart LR + subgraph "Endpoint Settings" + direction TB + StopEndpoint["Stop Endpoint"] + RestartEndpoint["Restart Endpoint"] + UpdateResource["Update Cloud Resource"] + UpdateFramework["Update Runtime Framework"] + UpdateEngineArgs["Update Engine Arguments"] + UpdateMaxReplica["Update Max Replica"] + UpdateMinReplica["Update Min Replica"] + UpdateVisibility["Update Visibility"] + DeleteEndpoint["Delete Endpoint"] + end + + User[["User"]] --> StopEndpoint & RestartEndpoint & UpdateResource & UpdateFramework & UpdateEngineArgs & UpdateMaxReplica & UpdateMinReplica & UpdateVisibility & DeleteEndpoint + + StopEndpoint & RestartEndpoint & UpdateResource & UpdateFramework & UpdateEngineArgs & UpdateMaxReplica & UpdateMinReplica & UpdateVisibility --> API["Backend API"] + DeleteEndpoint --> Confirmation["Confirmation Dialog"] + Confirmation --> API +``` + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:44-276]() + +#### 3.3.1 Updating Cloud Resources + +You can change the hardware resources allocated to your endpoint. This is useful for scaling up or down based on your performance needs and budget. + +1. Navigate to the endpoint settings page. +2. Ensure the endpoint is in the "Stopped" state. +3. Select a new cloud resource from the dropdown. +4. The change will be applied when you restart the endpoint. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:44-81]() + +#### 3.3.2 Updating Runtime Framework + +You can change the runtime framework used by your endpoint: + +1. Navigate to the endpoint settings page. +2. Ensure the endpoint is in the "Stopped" state. +3. Select a new compatible runtime framework from the dropdown. +4. The change will be applied when you restart the endpoint. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:84-124]() + +#### 3.3.3 Updating Engine Arguments + +You can modify engine arguments to optimize performance: + +1. Navigate to the endpoint settings page. +2. Ensure the endpoint is in the "Stopped" state. +3. Expand the engine arguments section. +4. Modify the arguments as needed. +5. Click "Update" to save changes. +6. The changes will be applied when you restart the endpoint. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:111-124]() + +#### 3.3.4 Updating Replica Count + +You can adjust the minimum and maximum number of replicas for your endpoint: + +1. Navigate to the endpoint settings page. +2. Ensure the endpoint is in the "Stopped" state. +3. Select new values for minimum and maximum replicas. +4. The changes will be applied when you restart the endpoint. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:128-182]() + +#### 3.3.5 Changing Visibility + +You can change the visibility of your endpoint between public and private: + +1. Navigate to the endpoint settings page. +2. Ensure the endpoint is in the "Stopped" state. +3. Select the desired visibility option. +4. Confirm the change in the confirmation dialog. +5. The new visibility will be applied when you restart the endpoint. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:186-224]() + +### 3.4 Deleting an Endpoint + +To delete an endpoint: + +1. Navigate to the endpoint settings page. +2. Scroll to the "Delete Endpoint" section. +3. Enter the confirmation text (endpoint name/ID) in the text field. +4. Click the "I understand and confirm deletion" button. +5. The endpoint will be permanently deleted. + +**Note**: Deletion is irreversible and will permanently remove the endpoint and all its files. + +Sources: [frontend/src/components/endpoints/EndpointSettings.vue:227-276]() + +## 4. Viewing Endpoint Information + +The endpoint detail page provides information about your deployed endpoint, including: + +- Endpoint URL +- Model ID +- Parameters (visibility, replica count) +- Cloud resource details +- Endpoint status + +```mermaid +flowchart TD + EndpointDetailPage["EndpointPage.vue"] --> EndpointInformation["Endpoint Information Display"] + EndpointDetailPage --> EndpointStatus["Endpoint Status"] + EndpointDetailPage --> EndpointPlayground["Endpoint Playground"] + + EndpointInformation --> EndpointURL["Endpoint URL"] + EndpointInformation --> ModelID["Model ID"] + EndpointInformation --> Parameters["Parameters"] + EndpointInformation --> CloudResource["Cloud Resource"] + + EndpointStatus --> ReplicaList["Replica Status List"] + + EndpointPlayground --> TextGeneration["Text Generation"] + EndpointPlayground --> TextToImage["Text to Image"] + EndpointPlayground --> FeatureExtraction["Feature Extraction"] + EndpointPlayground --> ImageTextToText["Image Text to Text"] +``` + +Sources: [frontend/src/components/endpoints/EndpointPage.vue:1-73]() + +## 5. Using the Endpoint Playground + +If your endpoint is in the "Running" state and has a compatible task type (text-generation, text-to-image, feature-extraction, or image-text-to-text), you can test it using the endpoint playground. + +The playground provides a user-friendly interface to: + +1. Send requests to your endpoint +2. View the responses +3. Adjust parameters for inference +4. Test different inputs + +Sources: [frontend/src/components/endpoints/EndpointPage.vue:64-70](), [frontend/src/locales/en_js/endpoints.js:72-89]() + +## 6. Endpoint API Overview + +Endpoints expose a REST API that allows you to send inference requests programmatically. The API endpoint URL is displayed on the endpoint detail page. + +Key API endpoints for managing endpoints include: + +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/models/{modelId}/run` | POST | Create a new endpoint | +| `/models/{modelId}/run/{endpointId}` | PUT | Update endpoint configuration | +| `/models/{modelId}/run/{endpointId}/stop` | PUT | Stop an endpoint | +| `/models/{modelId}/run/{endpointId}/start` | PUT | Start or restart an endpoint | +| `/models/{modelId}/run/{endpointId}` | DELETE | Delete an endpoint | + +Sources: [frontend/src/components/endpoints/NewEndpoint.vue:550-552](), [frontend/src/components/endpoints/EndpointSettings.vue:398-424](), [frontend/src/components/endpoints/EndpointSettings.vue:485-518]() + +## 7. Comparison with Fine-tuning and Evaluation + +CSGHub provides three main types of computing resources: Endpoints, Fine-tuning, and Evaluation. Here's how they compare: + +| Feature | Endpoints | Fine-tuning | Evaluation | +|---------|-----------|-------------|------------| +| Purpose | Deploy models for inference | Train and refine models | Evaluate model performance | +| Resource Type | Inference-optimized | Training-optimized | Evaluation-optimized | +| Typical Duration | Long-running | Time-limited task | Time-limited task | +| Interaction | Real-time API access | No direct interaction | Results report | +| Management | Start/stop/update | Start/stop/delete | View results | + +For more details on fine-tuning and evaluation, see [Model Fine-tuning](#3.2) and [Model Settings and Evaluation](#3.3). + +Sources: [frontend/src/components/endpoints/NewEndpoint.vue](), [frontend/src/components/finetune/NewFinetune.vue](), [frontend/src/components/evaluations/NewEvaluation.vue]() \ No newline at end of file diff --git a/docs/development/8_Model_Fine-tuning.md b/docs/development/8_Model_Fine-tuning.md new file mode 100644 index 000000000..187f6ccd6 --- /dev/null +++ b/docs/development/8_Model_Fine-tuning.md @@ -0,0 +1,269 @@ +# Model Fine-tuning + +Model Fine-tuning in CSGHub provides a structured approach to adapt pre-trained large language models (LLMs) to specific use cases. This document explains the fine-tuning system architecture, the process for creating and managing fine-tuning tasks, configuration options, and integration points. + +For information about deploying fine-tuned models, see [Creating and Managing Endpoints](#3.1). For details on evaluating model performance, see [Model Settings and Evaluation](#3.3). + +## 1. Fine-tuning System Architecture + +The fine-tuning system in CSGHub enables users to customize pre-trained models with domain-specific data. The system integrates with the platform's resource management, cluster configuration, and model registry components. + +### Architecture Overview + +```mermaid +flowchart TD + subgraph "Fine-tuning System" + NewFinetune["NewFinetune Component"] + FinetuneSettings["FinetuneSettings Component"] + EngineArgs["EngineArgs Component"] + end + + subgraph "Backend API" + ModelsAPI["Models API"] + ClusterAPI["Cluster API"] + ResourceAPI["Resource API"] + RuntimeFrameworkAPI["RuntimeFramework API"] + end + + subgraph "Infrastructure" + Clusters["Compute Clusters"] + Resources["Compute Resources"] + Models["Model Registry"] + end + + User(["User"]) --> NewFinetune + User --> FinetuneSettings + + NewFinetune --> ModelsAPI + NewFinetune --> ClusterAPI + NewFinetune --> ResourceAPI + NewFinetune --> RuntimeFrameworkAPI + NewFinetune --> EngineArgs + + FinetuneSettings --> ModelsAPI + FinetuneSettings --> ResourceAPI + FinetuneSettings --> EngineArgs + + ModelsAPI --> Models + ClusterAPI --> Clusters + ResourceAPI --> Resources + + classDef component fill:#f5f5f5,stroke:#333,stroke-width:1px; + class NewFinetune,FinetuneSettings,EngineArgs component; +``` + +Sources: [frontend/src/components/finetune/NewFinetune.vue:1-375](), [frontend/src/components/finetune/FinetuneSettings.vue:1-386](), [frontend/src/components/endpoints/EngineArgs.vue:1-127]() + +### Fine-tuning Creation Process + +```mermaid +sequenceDiagram + participant User + participant NewFinetune as "NewFinetune Component" + participant API as "Backend API" + participant Resources as "Compute Resources" + + User->>NewFinetune: Initialize fine-tuning task + NewFinetune->>API: Fetch available models (deploy_type=2) + API-->>NewFinetune: Return fine-tunable models + + User->>NewFinetune: Select model + NewFinetune->>API: Fetch runtime frameworks + API-->>NewFinetune: Return compatible frameworks + + User->>NewFinetune: Select cluster + NewFinetune->>API: Fetch available resources + API-->>NewFinetune: Return compatible resources + + User->>NewFinetune: Configure and submit + NewFinetune->>API: POST /models/{modelId}/finetune + API->>Resources: Allocate resources + API-->>NewFinetune: Return finetune ID + + NewFinetune->>User: Redirect to finetune detail page +``` + +Sources: [frontend/src/components/finetune/NewFinetune.vue:168-367]() + +## 2. Creating a Fine-tuning Task + +### Required Parameters + +To create a fine-tuning task, the following parameters must be specified: + +| Parameter | Description | Required | +|-----------|-------------|----------| +| deploy_name | Name for the fine-tuning task | Yes | +| model_id | Source model in format `owner/model_name` | Yes | +| cluster_id | Compute cluster for execution | Yes | +| resource_id | Computational resources for the task | Yes | +| runtime_framework_id | Framework for fine-tuning | Yes | +| secure_level | Visibility setting (1=public, 2=private) | Yes | + +Sources: [frontend/src/components/finetune/NewFinetune.vue:184-191]() + +### Model Selection Interface + +The system provides an autocomplete interface to select from available models that support fine-tuning. Only models that are compatible with the fine-tuning process (deploy_type=2) are displayed. + +```mermaid +flowchart LR + ModelInput["Model Input Field"] -->|"autocomplete"| ModelAPI["/runtime_framework/models?deploy_type=2"] + ModelAPI -->|"returns"| CompatibleModels["Compatible Models List"] + CompatibleModels -->|"selection"| FrameworkAPI["/models/{modelId}/runtime_framework?deploy_type=2"] + FrameworkAPI -->|"returns"| FrameworkOptions["Compatible Frameworks"] +``` + +Sources: [frontend/src/components/finetune/NewFinetune.vue:320-331]() + +### Resource and Framework Selection + +Resource selection follows a hierarchical process: + +1. Select a cluster/region for execution +2. View available computational resources in that cluster +3. Select a compatible framework based on the model and resource type + +The system filters available frameworks based on the selected resource type to ensure compatibility. + +Sources: [frontend/src/components/finetune/NewFinetune.vue:258-269](), [frontend/src/components/finetune/NewFinetune.vue:295-306]() + +## 3. Managing Fine-tuning Tasks + +### Fine-tuning Task Lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Creating + Creating --> Building: Submit task + Building --> Running: Initialization completes + Running --> Stopped: Stop command + Stopped --> Running: Start command + Running --> Completed: Task finishes + Running --> Failed: Error occurs + Stopped --> [*]: Delete task + Completed --> [*]: Delete task + Failed --> [*]: Delete task +``` + +Sources: [frontend/src/components/finetune/FinetuneSettings.vue:256-276]() + +### Starting and Stopping Tasks + +Fine-tuning tasks can be started and stopped through the FinetuneSettings interface. This allows users to pause resource-intensive operations and resume them later. + +The controls become available based on the current status of the fine-tuning task: +- Stop button is only active for initialized or running tasks +- Start button is only active for stopped tasks + +Sources: [frontend/src/components/finetune/FinetuneSettings.vue:51-85](), [frontend/src/components/finetune/FinetuneSettings.vue:284-298]() + +### Deleting Fine-tuning Tasks + +To delete a fine-tuning task, users must confirm by typing the full task identifier in the format `finetuneName/finetuneId`. This prevents accidental deletion of valuable fine-tuning tasks. + +Sources: [frontend/src/components/finetune/FinetuneSettings.vue:166-214](), [frontend/src/components/finetune/FinetuneSettings.vue:334-377]() + +## 4. Configuration Options + +### Resource Types + +Fine-tuning requires specific resource types, typically GPU-accelerated instances. The platform filters available resources to only show those compatible with fine-tuning operations (deploy_type=2). + +Each resource includes: +- Name and specifications +- Availability status +- Hardware type (e.g., GPU, CPU) +- Region information + +Sources: [frontend/src/components/finetune/NewFinetune.vue:258-269]() + +### Runtime Frameworks + +Runtime frameworks define the software stack used for fine-tuning. Available frameworks are filtered based on: +1. The selected model (only compatible frameworks are shown) +2. The selected resource type (frameworks must match the compute type) + +The system dynamically updates available frameworks when users change their resource selection. + +Sources: [frontend/src/components/finetune/NewFinetune.vue:280-293](), [frontend/src/components/finetune/NewFinetune.vue:295-306]() + +### Engine Arguments + +Advanced configuration is available through engine arguments, which allow fine-grained control over the fine-tuning process. These arguments vary based on the selected framework and may include: + +| Argument Type | Examples | Input Type | +|---------------|----------|------------| +| Numerical Parameters | block-size, gpu-memory-utilization | Select/Text | +| Data Type Options | dtype (float16, bfloat16, etc.) | Select | +| Feature Toggles | enable-prefix-caching, enforce-eager | Switch | +| Load Format | auto, pt, safetensors, etc. | Select | + +Sources: [frontend/src/components/endpoints/EngineArgs.vue:1-127](), [frontend/src/packs/useEngineArgs.js:1-54]() + +## 5. API Integration Points + +### Fine-tuning API Endpoints + +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/models/{modelId}/finetune` | POST | Create new fine-tuning task | +| `/models/{modelId}/finetune/{finetuneId}` | DELETE | Delete fine-tuning task | +| `/models/{modelId}/finetune/{finetuneId}/start` | PUT | Start/resume fine-tuning | +| `/models/{modelId}/finetune/{finetuneId}/stop` | PUT | Stop/pause fine-tuning | +| `/cluster` | GET | List available clusters | +| `/runtime_framework/models?deploy_type=2` | GET | List fine-tunable models | +| `/models/{modelId}/runtime_framework?deploy_type=2` | GET | List frameworks for model | + +Sources: [frontend/src/components/finetune/NewFinetune.vue:353-366](), [frontend/src/components/finetune/FinetuneSettings.vue:284-298](), [frontend/src/components/finetune/FinetuneSettings.vue:334-352]() + +### Fine-tuning Request Format + +The following JSON structure is used when creating a new fine-tuning task: + +```json +{ + "deploy_name": "my-finetune-task", + "model_id": "organization/model_name", + "cluster_id": "cluster_identifier", + "resource_id": "resource_id/order_detail_id", + "runtime_framework_id": "framework_id", + "secure_level": 2 +} +``` + +Sources: [frontend/src/components/finetune/NewFinetune.vue:184-191](), [frontend/src/components/finetune/NewFinetune.vue:348-353]() + +## 6. Integration with Other Systems + +Fine-tuning in CSGHub is closely integrated with other platform components: + +```mermaid +flowchart TD + subgraph "Fine-tuning System" + FinetuneComponent["Fine-tuning Interface"] + end + + subgraph "Related Systems" + ModelRegistry["Model Registry"] + ResourceConsole["Resource Console"] + Endpoints["Endpoint System"] + Evaluation["Model Evaluation"] + end + + ModelRegistry -->|"Source models"| FinetuneComponent + FinetuneComponent -->|"Creates fine-tuned"| ModelRegistry + FinetuneComponent -->|"Managed in"| ResourceConsole + FinetuneComponent -->|"Deploy via"| Endpoints + FinetuneComponent -->|"Evaluate with"| Evaluation + + classDef primary fill:#f5f5f5,stroke:#333,stroke-width:1px; + class FinetuneComponent primary; +``` + +- **Model Registry**: Source models are selected from the registry, and fine-tuned models are stored back to it +- **Resource Console**: All fine-tuning tasks are managed through the resource console +- **Endpoints**: Fine-tuned models can be deployed through the endpoint system +- **Evaluation**: Models can be evaluated before and after fine-tuning to measure improvement + +Sources: [frontend/src/components/finetune/NewFinetune.vue:363-365](), [frontend/src/components/finetune/FinetuneSettings.vue:349-351]() \ No newline at end of file diff --git a/docs/development/9_Model_Settings_and_Evaluation.md b/docs/development/9_Model_Settings_and_Evaluation.md new file mode 100644 index 000000000..7c8696e67 --- /dev/null +++ b/docs/development/9_Model_Settings_and_Evaluation.md @@ -0,0 +1,407 @@ +# Model Settings and Evaluation + +This technical documentation covers the model settings and evaluation capabilities in CSGHub. It explains how users can manage model metadata, configure visibility settings, and create/review model evaluations. For information about inference endpoints, see [Creating and Managing Endpoints](#3.1), and for details on model fine-tuning, see [Model Fine-tuning](#3.2). + +## 1. Model Settings Architecture + +The model settings interface allows administrators to manage key model properties, including names, descriptions, tags, and visibility. The system follows a component-based architecture where UI elements interact with backend APIs through fetch operations. + +```mermaid +graph TD + User(["User"])-->|"Updates"|ModelSettingsUI["ModelSettings Component"] + ModelSettingsUI-->|"Calls"|UpdateMethods["Update Methods"] + + subgraph "Update Methods" + UpdateNickname["updateNickname()"] + UpdateDesc["updateModelDesc()"] + UpdateTags["updateTags()"] + UpdateVisibility["changeVisibility()"] + DeleteModel["deleteModel()"] + end + + UpdateNickname-->ModelAPI["Model API"] + UpdateDesc-->ModelAPI + UpdateVisibility-->ModelAPI + + UpdateTags-->README["README.md Processing"] + README-->|"parseMD()"|MetadataExtract["Extract Metadata"] + MetadataExtract-->|"yaml.dump()"|MetadataUpdate["Update Metadata"] + MetadataUpdate-->READMEApi["README API"] + + DeleteModel-->DeleteAPI["Delete API"] + + ModelAPI-->RefreshDetail["fetchRepoDetail()"] + READMEApi-->RefreshDetail + DeleteAPI-->Redirect["Redirect to models list"] + + RefreshDetail-->ModelSettingsUI +``` + +The `ModelSettings.vue` component provides a comprehensive interface for managing model metadata and settings. It communicates with backend APIs to fetch and update model information. + +Sources: +- [frontend/src/components/models/ModelSettings.vue:1-286](). The main model settings component. +- [frontend/src/components/models/ModelSettings.vue:289-681](). Logic implementation for model settings. + +## 2. Model Metadata Management + +### 2.1 Basic Metadata Fields + +The ModelSettings component allows editing the following model metadata: + +| Field | Description | Update Method | API Endpoint | +|-------|-------------|---------------|--------------| +| Model Path | Namespace/model name (read-only) | N/A | N/A | +| Nickname | User-friendly model name | updateNickname() | /models/{path} | +| Description | Model description text | updateModelDesc() | /models/{path} | +| Visibility | Public or Private | changeVisibility() | /models/{path} | + +Each field has its own update method that validates input and makes the necessary API calls to update the model metadata. + +Sources: +- [frontend/src/components/models/ModelSettings.vue:27-49](). Nickname field and update functionality. +- [frontend/src/components/models/ModelSettings.vue:53-76](). Description field and update functionality. +- [frontend/src/components/models/ModelSettings.vue:202-242](). Visibility settings implementation. + +### 2.2 Tag Management System + +CSGHub implements a dual-storage tag system where different tag categories are stored in different locations: + +```mermaid +graph TD + subgraph "Tag Categories" + TaskTags["Task Tags"] + IndustryTags["Industry Tags"] + OtherTags["Other Tags"] + end + + subgraph "Storage Locations" + READMEMetadata["README.md Metadata"] + APIStorage["Industry Tags API"] + end + + TaskTags-->READMEMetadata + OtherTags-->READMEMetadata + IndustryTags-->APIStorage + + subgraph "User Interactions" + AddTag["Add Tag"] + RemoveTag["Remove Tag"] + SearchTag["Search Tag"] + end + + AddTag-->|"selectTag()"|UpdateSelectedTags["Update selectedTags[]"] + RemoveTag-->|"removeTag()"|UpdateSelectedTags + SearchTag-->|"showTagList()"|FilteredTags["Show Filtered Tags"] + + UpdateSelectedTags-->SaveChanges["Save Changes"] + SaveChanges-->|"updateTags()"|UpdateProcessing["Update Processing"] + + UpdateProcessing-->|"Task/Other Tags"|UpdateTagsInReadme["updateTagsInReadme()"] + UpdateProcessing-->|"Industry Tags"|UpdateIndustryTagsAPI["updateIndustryTagsAPI()"] + + UpdateTagsInReadme-->FetchREADME["Fetch README.md"] + FetchREADME-->ParseMetadata["Parse Metadata"] + ParseMetadata-->UpdateMetadata["Update Metadata"] + UpdateMetadata-->SaveREADME["Save README.md"] + + UpdateIndustryTagsAPI-->FetchRepoDetail["Refresh Model Data"] + SaveREADME-->FetchRepoDetail +``` + +The tag management system provides interfaces for both regular tags (stored in README.md) and industry tags (stored via API): + +1. **Task & Other Tags**: + - Stored in the model's README.md metadata section + - Updated via the updateTagsInReadme() method + - Requires parsing and updating the README.md file + +2. **Industry Tags**: + - Stored via a dedicated API endpoint + - Updated via the updateIndustryTagsAPI() method + - Directly linked to the model + +Sources: +- [frontend/src/components/models/ModelSettings.vue:80-134](). Task tags implementation. +- [frontend/src/components/models/ModelSettings.vue:138-198](). Industry tags implementation. +- [frontend/src/components/models/ModelSettings.vue:539-549](). updateTags() method. +- [frontend/src/components/models/ModelSettings.vue:574-608](). updateTagsInReadme() and updateReadme() methods. +- [frontend/src/components/models/ModelSettings.vue:609-624](). updateIndustryTagsAPI() method. + +### 2.3 Visibility Controls + +The visibility system allows toggling between private and public model access: + +```mermaid +graph TD + User(["User"])-->VisibilityToggle["Visibility Toggle"] + VisibilityToggle-->|"changeVisibility()"|ConfirmationDialog["Confirmation Dialog"] + ConfirmationDialog-->|"Confirmed"|ChangeVisibilityCall["changeVisibilityCall()"] + ConfirmationDialog-->|"Cancelled"|MessageCancel["Cancel Message"] + ChangeVisibilityCall-->|"updateModel()"|APIUpdate["API Update"] + APIUpdate-->|"updateVisibility()"|StoreUpdate["Store Update"] + StoreUpdate-->|"fetchRepoDetail()"|UIRefresh["UI Refresh"] +``` + +The visibility toggle interface provides a dropdown with "Public" and "Private" options. When changed, the system confirms the change with the user before updating the model visibility status. + +Sources: +- [frontend/src/components/models/ModelSettings.vue:202-242](). Visibility settings UI. +- [frontend/src/components/models/ModelSettings.vue:499-532](). changeVisibility() method. +- [frontend/src/components/models/ModelSettings.vue:534-538](). changeVisibilityCall() method. +- [frontend/src/components/models/ModelSettings.vue:650-666](). updateModel() method. + +### 2.4 Model Deletion + +Model deletion includes a safety mechanism requiring the user to type the full model path to confirm: + +```mermaid +graph TD + User(["User"])-->DeleteSection["Delete Section"] + DeleteSection-->|"Input Model Path"|ValidateInput["Validate Input"] + ValidateInput-->|"Match"|EnableDelete["Enable Delete Button"] + EnableDelete-->|"clickDelete()"|DeleteConfirm["Delete Confirmation"] + DeleteConfirm-->|"deleteModel()"|DeleteAPI["Delete API Call"] + DeleteAPI-->|"Success"|SuccessMessage["Success Message"] + SuccessMessage-->Redirect["Redirect to Models List"] +``` + +This multi-step confirmation process helps prevent accidental deletion of models. + +Sources: +- [frontend/src/components/models/ModelSettings.vue:244-285](). Model deletion UI. +- [frontend/src/components/models/ModelSettings.vue:401-410](). clickDelete() method. +- [frontend/src/components/models/ModelSettings.vue:484-497](). deleteModel() method. + +## 3. Model Evaluation System + +CSGHub includes a comprehensive evaluation system for assessing model performance. The evaluation table provides a centralized view of all evaluations. + +### 3.1 Evaluation Table Architecture + +```mermaid +graph TD + subgraph "Evaluation Table Component" + TableDisplay["Evaluation Table"] + Pagination["Pagination"] + ActionButtons["Action Buttons"] + end + + subgraph "Backend Integration" + FetchEvaluations["fetchEvaluations()"] + DeleteEvaluation["deleteEvaluation()"] + DetailEvaluation["detailEvaluation()"] + end + + subgraph "Status Components" + SuccessStatus["SuccessStatus (Completed)"] + WarningStatus["WarningStatus (Pending)"] + ErrorStatus["ErrorStatus (Failed)"] + BuildStatus["BuildStatus (Running)"] + end + + TableDisplay-->FetchEvaluations + ActionButtons-->DeleteEvaluation + ActionButtons-->DetailEvaluation + + FetchEvaluations-->|"response"|TableDisplay + TableDisplay-->|"render"|StatusComponents["Status Components"] + + StatusComponents-->SuccessStatus + StatusComponents-->WarningStatus + StatusComponents-->ErrorStatus + StatusComponents-->BuildStatus + + Pagination-->|"currentChange"|FetchEvaluations +``` + +The evaluation table provides a unified interface for viewing and managing model evaluations across different datasets. + +Sources: +- [frontend/src/components/resource_console/EvaluationTable.vue:1-142](). Evaluation table UI components. +- [frontend/src/components/resource_console/EvaluationTable.vue:229-242](). fetchEvaluations() method. +- [frontend/src/components/resource_console/EvaluationTable.vue:244-253](). deleteEvaluation() method. +- [frontend/src/components/resource_console/EvaluationTable.vue:255-259](). detailEvaluation() method. + +### 3.2 Evaluation Data Structure + +Each evaluation record contains the following key information: + +| Field | Description | Display Format | +|-------|-------------|----------------| +| task_name | Evaluation name | Text | +| repo_ids | Models being evaluated | Text list | +| datasets | Datasets used for evaluation | Text list with tooltips | +| submit_time | When the evaluation was created | Formatted date | +| status | Current status (Succeeded, Pending, Failed, Running) | Status component with icon | +| task_desc | Evaluation description | Text | +| download_url | URL for downloading results | Link (if available) | +| result_url | URL for viewing detailed results | Link (if available) | + +The table displays evaluation metadata and provides actions to download results, view details, or delete evaluations. + +Sources: +- [frontend/src/components/resource_console/EvaluationTable.vue:15-42](). Evaluation name and model display. +- [frontend/src/components/resource_console/EvaluationTable.vue:44-90](). Dataset information display. +- [frontend/src/components/resource_console/EvaluationTable.vue:92-139](). Status and actions display. +- [frontend/src/components/resource_console/EvaluationTable.vue:190-224](). Status mapping and components. + +### 3.3 Evaluation Status Workflow + +Evaluations progress through different status states during their lifecycle: + +```mermaid +graph LR + Creation["Creation"]-->Pending["Pending"] + Pending-->Running["Running"] + Running-->|"Success"|Succeeded["Succeeded"] + Running-->|"Error"|Failed["Failed"] + + subgraph "Status Components" + Pending-->WarningStatus["WarningStatus"] + Running-->BuildStatus["BuildStatus"] + Succeeded-->SuccessStatus["SuccessStatus"] + Failed-->ErrorStatus["ErrorStatus"] + end + + subgraph "Available Actions" + Succeeded-->|"Enable"|Download["Download Results"] + Succeeded-->|"Enable"|ViewDetails["View Details"] + Failed-->|"Enable"|Delete["Delete Evaluation"] + Pending-->|"Enable"|Delete + Running-->|"Enable"|Delete + Succeeded-->|"Enable"|Delete + end +``` + +The evaluation status determines which actions are available to the user, with completed evaluations enabling result viewing and downloading. + +Sources: +- [frontend/src/components/resource_console/EvaluationTable.vue:105-122](). Status component rendering. +- [frontend/src/components/resource_console/EvaluationTable.vue:148-176](). Action buttons based on status. +- [frontend/src/components/resource_console/EvaluationTable.vue:190-225](). Status mapping and component selection. + +## 4. Integration with Resource Console + +The Resource Console provides a unified interface for managing all resource types, including model evaluations. + +```mermaid +graph TD + ResourceConsole["ResourceConsoleIndex"]-->ResourceSections["Resource Sections"] + ResourceSections-->EvaluationSection["Evaluation Section"] + ResourceSections-->EndpointSection["Endpoint Section"] + ResourceSections-->FinetuneSection["Finetune Section"] + + EvaluationSection-->EvaluationTable["EvaluationTable"] + EvaluationTable-->FetchEvaluations["fetchEvaluations()"] + + FetchEvaluations-->APICall["API Call to /user/{username}/evaluations"] + APICall-->DisplayResults["Display Results"] + + EvaluationTable-->EvaluationActions["Evaluation Actions"] + EvaluationActions-->ViewDetails["View Details"] + EvaluationActions-->Download["Download Results"] + EvaluationActions-->Delete["Delete Evaluation"] +``` + +The Resource Console integrates evaluations with other resource types (endpoints, finetunes) to provide a comprehensive management interface. + +Sources: +- [frontend/src/components/resource_console/ResourceConsoleIndex.vue:47-57](). Evaluation section in Resource Console. +- [frontend/src/components/resource_console/EvaluationTable.vue:1-336](). Evaluation table component. + +## 5. Related UI Components + +Several UI components are used across both model settings and evaluation interfaces: + +### 5.1 Status Components + +Status components provide visual indicators for different evaluation states: + +| Component | Status | Visual Appearance | +|-----------|--------|-------------------| +| SuccessStatus | Succeeded | Green checkmark with "Succeeded" text | +| WarningStatus | Pending | Yellow warning icon with "Pending" text | +| ErrorStatus | Failed | Red error icon with "Failed" text | +| BuildStatus | Running | Blue spinner with "Running" text | + +These components maintain a consistent visual language across the application. + +Sources: +- [frontend/src/components/resource_console/EvaluationTable.vue:195-198](). Status component imports. +- [frontend/src/components/resource_console/EvaluationTable.vue:218-226](). Status component mapping. + +### 5.2 Pagination Component + +The CsgPagination component provides standardized pagination for evaluation tables: + +```mermaid +graph LR + EvaluationTable["EvaluationTable"]-->Pagination["CsgPagination"] + Pagination-->|"currentChange event"|FetchEvaluations["fetchEvaluations()"] + Pagination-->|"Parameters"|PaginationParams["perPage, currentPage, total"] +``` + +This component ensures consistent pagination across the application and handles interactions with the backend API. + +Sources: +- [frontend/src/components/resource_console/EvaluationTable.vue:180-186](). Pagination component in evaluation table. +- [frontend/src/components/resource_console/EvaluationTable.vue:229-242](). Pagination-aware fetchEvaluations method. + +## 6. Backend API Integration + +Both model settings and evaluation components interact with backend APIs to fetch and update data. + +### 6.1 Model Settings API Endpoints + +| Endpoint | Method | Purpose | Implementation | +|----------|--------|---------|----------------| +| `/models/{path}` | GET | Fetch model details | fetchRepoDetail() | +| `/models/{path}` | PUT | Update model metadata | updateModel() | +| `/models/{path}` | DELETE | Delete model | deleteModel() | +| `/models/{path}/blob/README.md` | GET | Fetch README content | fetchReadme() | +| `/models/{path}/raw/README.md` | PUT | Update README content | updateReadme() | +| `/models/{path}/tags/industry` | POST | Update industry tags | updateIndustryTagsAPI() | +| `/tags` | GET | Fetch available tags | getIndustryTags() | + +Sources: +- [frontend/src/components/models/ModelSettings.vue:484-497](). deleteModel() implementation. +- [frontend/src/components/models/ModelSettings.vue:565-573](). fetchReadme() implementation. +- [frontend/src/components/models/ModelSettings.vue:586-608](). updateReadme() implementation. +- [frontend/src/components/models/ModelSettings.vue:609-624](). updateIndustryTagsAPI() implementation. +- [frontend/src/components/models/ModelSettings.vue:650-666](). updateModel() implementation. + +### 6.2 Evaluation API Endpoints + +| Endpoint | Method | Purpose | Implementation | +|----------|--------|---------|----------------| +| `/user/{username}/evaluations` | GET | Fetch user evaluations | fetchEvaluations() | +| `/evaluations/{id}` | DELETE | Delete evaluation | deleteEvaluation() | +| `/evaluations/{id}` | GET | View evaluation details | detailEvaluation() | + +Sources: +- [frontend/src/components/resource_console/EvaluationTable.vue:229-242](). fetchEvaluations() implementation. +- [frontend/src/components/resource_console/EvaluationTable.vue:244-253](). deleteEvaluation() implementation. +- [frontend/src/components/resource_console/EvaluationTable.vue:255-259](). detailEvaluation() implementation. + +## 7. Internationalization Support + +Both model settings and evaluation interfaces support internationalization (i18n) with English and Chinese translations: + +| Feature | English Key | Chinese Implementation | +|---------|-------------|------------------------| +| Model Settings | models.edit.* | Translated strings for all UI elements | +| Evaluation Table | evaluation.list.* | Translated strings for column headers and actions | +| Status Labels | Direct mapping | statusMapping object maps English to Chinese | + +Sources: +- [frontend/src/components/models/ModelSettings.vue:8-12](). i18n for model name section. +- [frontend/src/components/resource_console/EvaluationTable.vue:211-216](). Status mapping for different languages. +- [frontend/src/locales/en_js/finetune.js:1-56](). English translations. +- [frontend/src/locales/zh_js/finetune.js:1-56](). Chinese translations. + +## Summary + +The Model Settings and Evaluation system in CSGHub provides comprehensive tools for managing model metadata and assessing model performance. The model settings component allows users to update model information, manage tags, and control visibility, while the evaluation system enables users to create, manage, and review model evaluations across different datasets. + +These components integrate seamlessly with the wider CSGHub platform, providing a consistent user experience and ensuring that model metadata and evaluation results are easily accessible to users. \ No newline at end of file From 0d78998cb240a3c8356936e94271c89150722815 Mon Sep 17 00:00:00 2001 From: zhendi Date: Fri, 30 May 2025 21:10:26 +0800 Subject: [PATCH 2/6] docs(i18n): remove unnecessary closing tag Removed the closing tag from the flowchart --- docs/development/16_Internationalization_i18n.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/development/16_Internationalization_i18n.md b/docs/development/16_Internationalization_i18n.md index c778df725..96254b9d7 100644 --- a/docs/development/16_Internationalization_i18n.md +++ b/docs/development/16_Internationalization_i18n.md @@ -25,7 +25,6 @@ flowchart TD I18n --- Components["Vue Components"] Components --- TransMethod["$t() Translation Method"] - ``` Sources: [frontend/src/locales/en_js/all.js](), [frontend/src/locales/zh_js/all.js](), [frontend/src/locales/en_js/endpoints.js](), [frontend/src/locales/zh_js/endpoints.js]() From 5a49da16036bcbf26d1d235bc680c4df6be91ead Mon Sep 17 00:00:00 2001 From: zhendi Date: Fri, 30 May 2025 21:15:10 +0800 Subject: [PATCH 3/6] fix(docs): update relative paths in documentation Updated relative paths in various sections: - CSGHub Overview - Core Components - Repository System - User and Authentication System - Resource Management - Internationalization - Deployment Options - Version History - Technical Implementation --- docs/development/1_CSGHub_Overview.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/development/1_CSGHub_Overview.md b/docs/development/1_CSGHub_Overview.md index 47a33e12f..871742365 100644 --- a/docs/development/1_CSGHub_Overview.md +++ b/docs/development/1_CSGHub_Overview.md @@ -47,7 +47,7 @@ flowchart TB StarhubServer --> S3 ``` -Sources: [README.md:12-14](frontend/src/components/navbar/Navbar.vue), [go.mod:1-19](go.mod), [config/config.go:14-40](config/config.go) +Sources: [README.md:12-14](../../frontend/src/components/navbar/Navbar.vue), [go.mod:1-19](../../go.mod), [config/config.go:14-40](../../config/config.go) ## Core Components @@ -121,7 +121,7 @@ flowchart TB StarhubServer --> FinetuneMgmt ``` -Sources: [frontend/src/main.js:1-167](frontend/src/main.js), [config/config.go:14-40](config/config.go), [README.md:12-18](README.md) +Sources: [frontend/src/main.js:1-167](../../frontend/src/main.js), [config/config.go:14-40](../../config/config.go), [README.md:12-18](../../README.md) ## Repository System @@ -197,7 +197,7 @@ flowchart TB Spaces --> VersionControl ``` -Sources: [frontend/src/main.js:32-51](frontend/src/main.js), [README.md:12-14](README.md), [frontend/src/components/navbar/Navbar.vue:159-195](frontend/src/components/navbar/Navbar.vue) +Sources: [frontend/src/main.js:32-51](../../frontend/src/main.js), [README.md:12-14](../../README.md), [frontend/src/components/navbar/Navbar.vue:159-195](../../frontend/src/components/navbar/Navbar.vue) ## User and Authentication System @@ -243,7 +243,7 @@ flowchart TB Session --> UserStore ``` -Sources: [frontend/src/stores/UserStore.js:1-95](frontend/src/stores/UserStore.js), [frontend/src/components/navbar/Navbar.vue:60-222](frontend/src/components/navbar/Navbar.vue), [frontend/src/packs/persistPinia.js:1-20](frontend/src/packs/persistPinia.js) +Sources: [frontend/src/stores/UserStore.js:1-95](../../frontend/src/stores/UserStore.js), [frontend/src/components/navbar/Navbar.vue:60-222](../../frontend/src/components/navbar/Navbar.vue), [frontend/src/packs/persistPinia.js:1-20](../../frontend/src/packs/persistPinia.js) ## Resource Management @@ -326,7 +326,7 @@ flowchart TB EvalRun --> EvalResults ``` -Sources: [frontend/src/main.js:45-58](frontend/src/main.js), [docs/csghub_saas_en.md:90-178](docs/csghub_saas_en.md), [docs/csghub_saas_zh.md:90-178](docs/csghub_saas_zh.md) +Sources: [frontend/src/main.js:45-58](../../frontend/src/main.js), [docs/csghub_saas_en.md:90-178](../../docs/csghub_saas_en.md), [docs/csghub_saas_zh.md:90-178](../../docs/csghub_saas_zh.md) ## Internationalization @@ -366,7 +366,7 @@ flowchart TB I18n --> Messages ``` -Sources: [frontend/src/main.js:132-139](frontend/src/main.js), [frontend/src/components/navbar/Navbar.vue:44-57](frontend/src/components/navbar/Navbar.vue) +Sources: [frontend/src/main.js:132-139](../../frontend/src/main.js), [frontend/src/components/navbar/Navbar.vue:44-57](../../frontend/src/components/navbar/Navbar.vue) ## Deployment Options @@ -379,7 +379,7 @@ CSGHub can be deployed in several ways: | Helm | Kubernetes-based deployment | Production, scalable environments | | SaaS | Cloud service at opencsg.com | Immediate access without infrastructure | -Sources: [README.md:39-47](README.md), [docs/release_notes.md:12-15](docs/release_notes.md) +Sources: [README.md:39-47](../../README.md), [docs/release_notes.md:12-15](../../docs/release_notes.md) ## Version History @@ -391,9 +391,9 @@ CSGHub follows a regular release cycle with version updates. Key milestones incl - v1.0.0 (2024.10.15): Stable multi-sync function, Docker for ARM, region support - v0.9.0 (2024.09.15): Migrated to Go, added Collections feature -For a complete history, see the [release notes](docs/release_notes.md). +For a complete history, see the [release notes](../../docs/release_notes.md). -Sources: [docs/release_notes.md:3-31](docs/release_notes.md) +Sources: [docs/release_notes.md:3-31](../../docs/release_notes.md) ## Technical Implementation @@ -450,7 +450,7 @@ flowchart TB Go --> Git ``` -Sources: [frontend/package.json:14-43](frontend/package.json), [go.mod:5-18](go.mod), [frontend/src/main.js:1-8](frontend/src/main.js) +Sources: [frontend/package.json:14-43](../../frontend/package.json), [go.mod:5-18](../../go.mod), [frontend/src/main.js:1-8](../../frontend/src/main.js) ## Conclusion From 1046551b697f3e89a9065169482f4337f1c7d145 Mon Sep 17 00:00:00 2001 From: zhendi Date: Sat, 31 May 2025 17:16:56 +0800 Subject: [PATCH 4/6] fix(docs): remove unnecessary source links - Cleaned up source links in documentation - Removed empty source links from multiple sections - Updated user authentication and session management docs - Revised repository management documentation --- .../10_User_and_Authentication_System.md | 30 ++++----- ...r_Authentication_and_Session_Management.md | 39 ------------ .../12_User_Profile_and_Settings.md | 48 --------------- .../13_Organizations_and_Collections.md | 24 ++++---- docs/development/14_Frontend_Architecture.md | 32 +++++----- .../15_Component_System_and_Navigation.md | 27 ++++---- .../16_Internationalization_i18n.md | 20 +++--- .../17_API_Integration_and_Data_Fetching.md | 34 ----------- docs/development/18_Admin_Panel.md | 58 ------------------ docs/development/19_Backend_Architecture.md | 26 ++++---- docs/development/1_CSGHub_Overview.md | 17 ------ docs/development/20_Routing_and_Middleware.md | 28 ++++----- docs/development/21_Repository_Handlers.md | 18 +++--- .../22_Development_and_Deployment.md | 29 +++++---- .../2_Repository_Management_System.md | 60 ------------------ .../3_Repository_Detail_Components.md | 24 ++------ .../4_Repository_Clone_and_Actions.md | 26 ++++---- .../5_Repository_Browsing_and_Filtering.md | 26 ++++---- .../6_Model_Deployment_and_Fine-tuning.md | 52 ---------------- .../7_Creating_and_Managing_Endpoints.md | 35 ++++++----- docs/development/8_Model_Fine-tuning.md | 27 ++++---- .../9_Model_Settings_and_Evaluation.md | 61 ------------------- 22 files changed, 178 insertions(+), 563 deletions(-) diff --git a/docs/development/10_User_and_Authentication_System.md b/docs/development/10_User_and_Authentication_System.md index 5d8516b8a..3aed2721d 100644 --- a/docs/development/10_User_and_Authentication_System.md +++ b/docs/development/10_User_and_Authentication_System.md @@ -48,7 +48,7 @@ flowchart TD AuthMiddleware --> UserModel ``` -Sources: [frontend/src/components/navbar/Navbar.vue](), [frontend/src/stores/UserStore.js](), [internal/handlers/render/session.go](), [internal/models/user.go]() + ## 2. User Model and Database Schema @@ -81,7 +81,7 @@ classDiagram } ``` -Sources: [internal/models/user.go:19-31](), [pkg/database/migrations/20240902082008_create_admin_photo.go:9-22]() + ### 2.2. Role-Based Access Control @@ -100,7 +100,7 @@ The `User` struct provides methods to check and manage roles: - `IsAdmin()`: Checks if user has admin privileges - `IsSuperUser()`: Checks if user has super user privileges -Sources: [internal/models/user.go:33-94]() + ## 3. Authentication Flow @@ -134,7 +134,7 @@ sequenceDiagram Frontend->>UserStore: Initialize User Data ``` -Sources: [internal/handlers/render/session.go:71-151]() + ### 3.1. Login Process @@ -146,7 +146,7 @@ Sources: [internal/handlers/render/session.go:71-151]() 6. User session information is stored in cookies 7. User is redirected to the home page or a previously accessed page -Sources: [internal/handlers/render/session.go:47-69](), [internal/handlers/render/session.go:71-151]() + ### 3.2. Logout Process @@ -155,7 +155,7 @@ Sources: [internal/handlers/render/session.go:47-69](), [internal/handlers/rende 3. All cookies are removed, including login tokens 4. User is redirected to the home page -Sources: [frontend/src/components/navbar/Navbar.vue:214-222](), [frontend/src/components/navbar/Navbar.vue:379-382](), [internal/handlers/render/session.go:51-65]() + ## 4. Session Management @@ -173,7 +173,7 @@ The system uses several cookies to manage user sessions: | `previous_path` | Stores the last visited path for redirection after login | Session | | `locale` | Stores the user's language preference | Persistent | -Sources: [internal/handlers/render/session.go:141-143](), [frontend/src/components/navbar/Navbar.vue:351-352]() + ### 4.2. Frontend State Persistence @@ -184,7 +184,7 @@ The frontend uses Pinia with a custom persistence plugin to maintain user state 3. If valid, the user state is restored; otherwise, it's cleared 4. This persistence layer works alongside cookies to provide a seamless user experience -Sources: [frontend/src/packs/persistPinia.js:1-20](), [frontend/src/stores/UserStore.js:89-93]() + ## 5. User Store and Frontend Integration @@ -218,7 +218,7 @@ flowchart TD ComputedProps --> |"Controls UI visibility"| Navbar ``` -Sources: [frontend/src/stores/UserStore.js:7-93](), [frontend/src/components/navbar/Navbar.vue:322-323](), [frontend/src/components/navbar/Navbar.vue:361]() + ### 5.1. Key Features of the User Store @@ -238,7 +238,7 @@ Sources: [frontend/src/stores/UserStore.js:7-93](), [frontend/src/components/nav - Short expiration time (2 minutes) to ensure data freshness - Synchronized with server on page load -Sources: [frontend/src/stores/UserStore.js:7-93](), [frontend/src/packs/persistPinia.js:1-20]() + ## 6. User Interface Components @@ -254,7 +254,7 @@ The Navbar component is the main entry point for user authentication interaction 4. Handles logout process 5. Displays notifications for missing email or required username changes -Sources: [frontend/src/components/navbar/Navbar.vue:1-443]() + ### 6.2. Profile Management Components @@ -270,7 +270,7 @@ Several components handle user profile management: These components are registered in the main application entry point and rendered based on the current route. -Sources: [frontend/src/main.js:26-28](), [frontend/src/main.js:30-31]() + ## 7. System Configuration and Initialization @@ -284,7 +284,7 @@ The authentication system is initialized during application startup. 4. Routes are initialized including authentication middleware 5. The HTTP server is started to handle requests -Sources: [cmd/csghub-portal/cmd/start/server.go:15-51](), [internal/models/common.go:15-22]() + ### 7.2. Frontend Initialization @@ -295,7 +295,7 @@ Sources: [cmd/csghub-portal/cmd/start/server.go:15-51](), [internal/models/commo 5. Router is configured for navigation 6. The application is mounted to the DOM -Sources: [frontend/src/main.js:1-168](), [frontend/src/admin.js:1-71]() + ## 8. Admin User Management @@ -306,7 +306,7 @@ The system includes specialized admin interfaces for user management: 3. System configuration settings 4. Additional management features for admin users -Sources: [frontend/src/main.js:61-69](), [frontend/src/components/navbar/Navbar.vue:137-145]() + ## Summary diff --git a/docs/development/11_User_Authentication_and_Session_Management.md b/docs/development/11_User_Authentication_and_Session_Management.md index 7f92f8522..37153a5a2 100644 --- a/docs/development/11_User_Authentication_and_Session_Management.md +++ b/docs/development/11_User_Authentication_and_Session_Management.md @@ -29,10 +29,6 @@ flowchart TD M --> D ``` -Sources: -- [frontend/src/components/navbar/Navbar.vue:60-225]() -- [internal/handlers/render/session.go:18-151]() -- [frontend/src/stores/UserStore.js:1-95]() ## 2. Authentication Flow @@ -69,9 +65,6 @@ sequenceDiagram UI->>UI: Initialize UserStore ``` -Sources: -- [internal/handlers/render/session.go:47-49]() - Login handler -- [internal/handlers/render/session.go:71-151]() - Session creation and JWT verification ### 2.2 JWT Token Handling @@ -86,9 +79,6 @@ The token is stored in cookies: - `login_identity`: Contains the user's UUID - `can_change_username`: Indicates if the user needs to update their username -Sources: -- [internal/handlers/render/session.go:80-93]() - JWT token verification -- [internal/handlers/render/session.go:141-143]() - Setting cookies with user information ## 3. Session Management @@ -126,9 +116,6 @@ classDiagram SessionHandlerImpl --> UserStore : "manages user data for" ``` -Sources: -- [internal/handlers/render/session.go:18-29]() - Cookie constants and SessionHandler interface -- [frontend/src/stores/UserStore.js:1-95]() - Frontend session state management ### 3.2 Logout Process @@ -137,9 +124,6 @@ The logout process: 2. Clears the UserStore state using the clearStore action 3. Redirects the user to the home page or a specified redirect path -Sources: -- [internal/handlers/render/session.go:51-65]() - Backend logout handler -- [frontend/src/components/navbar/Navbar.vue:379-382]() - Frontend logout function ### 3.3 Session State Management @@ -149,9 +133,6 @@ The frontend uses Pinia for state management with the following features: - **State Properties**: User details, roles, login status, and action limitations - **Computed Properties**: Authorization flags like `isAdmin`, `isLoggedIn`, `actionLimited` -Sources: -- [frontend/src/stores/UserStore.js:7-93]() - UserStore definition -- [frontend/src/packs/persistPinia.js:1-20]() - State persistence implementation ## 4. User Model and Roles @@ -176,9 +157,6 @@ erDiagram } ``` -Sources: -- [internal/models/user.go:19-31]() - User struct definition -- [pkg/database/migrations/20240902082008_create_admin_photo.go:9-22]() - User database migration ### 4.2 Role-Based Access Control @@ -198,10 +176,6 @@ Role checks are implemented in both backend and frontend: The frontend UI adapts based on the user's roles, showing or hiding specific menu items and functionality. -Sources: -- [internal/models/user.go:33-94]() - Role definitions and helper methods -- [frontend/src/stores/UserStore.js:26-33]() - Frontend role computation -- [frontend/src/components/navbar/Navbar.vue:137-145]() - Admin panel access control ## 5. Frontend Implementation Details @@ -223,9 +197,6 @@ The component also displays warning banners when: - Username needs to be changed - Both email and username need attention -Sources: -- [frontend/src/components/navbar/Navbar.vue:60-225]() - Avatar dropdown implementation -- [frontend/src/components/navbar/Navbar.vue:287-315]() - Warning banners ### 5.2 User Store Integration @@ -247,9 +218,6 @@ flowchart TD C --> I["Authorization checks"] ``` -Sources: -- [frontend/src/main.js:72-73]() - Pinia initialization with persistence plugin -- [frontend/src/components/navbar/Navbar.vue:371-377]() - User fetching and store initialization ## 6. Security Considerations @@ -259,9 +227,6 @@ Sources: - Role checks are performed on both client and server sides - Action limitations are implemented for users with incomplete profiles -Sources: -- [internal/handlers/render/session.go:80-93]() - JWT verification process -- [frontend/src/packs/persistPinia.js:4-11]() - State expiration implementation ## 7. Session Initialization Process @@ -272,7 +237,3 @@ The session initialization follows these steps: - The Navbar component makes an API call to fetch user data - The UserStore is initialized with the fetched data 3. Protected routes and components check the UserStore to determine access rights - -Sources: -- [frontend/src/components/navbar/Navbar.vue:384-388]() - User data fetching on component mount -- [frontend/src/stores/UserStore.js:35-50]() - UserStore initialization logic \ No newline at end of file diff --git a/docs/development/12_User_Profile_and_Settings.md b/docs/development/12_User_Profile_and_Settings.md index 63537f3f0..754acbdf9 100644 --- a/docs/development/12_User_Profile_and_Settings.md +++ b/docs/development/12_User_Profile_and_Settings.md @@ -42,11 +42,6 @@ graph TD JWTToken -->|"updates"| UserStoreJS ``` -Sources: -- [frontend/src/components/user_settings/Profile.vue]() -- [frontend/src/components/user_settings/ProfileEdit.vue]() -- [frontend/src/components/user_settings/Menu.vue]() -- [frontend/src/components/popup/UpdateUsername.vue]() ### Profile Display Component @@ -86,10 +81,6 @@ graph TD ProfileData -->|"renders"| OrganizationSection ``` -Sources: -- [frontend/src/components/user_settings/Profile.vue:92-95]() - `isCurrentUser` computed property -- [frontend/src/components/user_settings/Profile.vue:100-114]() - `fetchUserInfo` method -- [frontend/src/components/user_settings/Profile.vue:43-66]() - Organization section ## 2. Profile Editing System @@ -140,11 +131,6 @@ sequenceDiagram ProfileEditVue->>User: Show success message ``` -Sources: -- [frontend/src/components/user_settings/ProfileEdit.vue:1-316]() -- [frontend/src/components/user_settings/ProfileEdit.vue:248-292]() - `updateProfile` function -- [frontend/src/components/user_settings/ProfileEdit.vue:294-311]() - `saveProfile` function -- [frontend/src/components/user_settings/ProfileEdit.vue:163-172]() - Email validation ### Avatar Management @@ -156,8 +142,6 @@ The `ProfileEdit.vue` component provides methods for avatar management: The avatar is uploaded to `/internal_api/upload` endpoint using `csrfFetch` to include CSRF protection. -Sources: -- [frontend/src/components/user_settings/ProfileEdit.vue:196-221]() - Avatar management methods ## 3. Username Management @@ -182,10 +166,6 @@ The username validation includes multiple rules: 5. Only allows letters, numbers, and specific special characters (-_.) 6. No consecutive special characters -Sources: -- [frontend/src/components/popup/UpdateUsername.vue:1-141]() -- [frontend/src/components/popup/UpdateUsername.vue:55-96]() - Username validation rules -- [frontend/src/components/user_settings/ProfileEdit.vue:226-246]() - Username update confirmation ## 4. Settings Navigation @@ -204,11 +184,6 @@ The menu offers different navigation options: 2. Access tokens (`/settings/access-token`) - only for non-limited users 3. SSH keys (`/settings/ssh-keys`) - only for non-limited users -Sources: -- [frontend/src/components/user_settings/Menu.vue:1-151]() -- [frontend/src/components/user_settings/Menu.vue:31-46]() - Menu links -- [frontend/src/components/user_settings/Menu.vue:51-64]() - Mobile tabs -- [frontend/src/components/user_settings/Menu.vue:69-100]() - Confirmation dialog ## 5. Integration with Other Systems @@ -260,11 +235,6 @@ graph TD AddToCollectionsVue -->|"displays"| CollectionCardsVue ``` -Sources: -- [frontend/src/components/organizations/OrganizationDetail.vue]() -- [frontend/src/components/collections/AddToCollections.vue]() -- [frontend/src/components/resource_console/ResourceConsoleIndex.vue]() -- [frontend/src/components/shared/ProfileRepoList.vue]() ### Organization Integration @@ -273,9 +243,6 @@ Users can belong to organizations, and the profile system displays organization - Organization role information (admin, write, read) affects available actions - Organization details are retrieved via `/organization/:name` endpoint -Sources: -- [frontend/src/components/user_settings/Profile.vue:43-66]() - Organization display section -- [frontend/src/components/organizations/OrganizationDetail.vue:66-68]() - Organization settings link ### Collections Integration @@ -284,9 +251,6 @@ The profile system allows users to add repositories to their collections: - The `AddToCollections.vue` component enables adding repositories to collections - Collection management is integrated with the user profile system via the UserStore -Sources: -- [frontend/src/components/collections/AddToCollections.vue:1-190]() -- [frontend/src/components/shared/ProfileRepoList.vue:17-27]() - Collections section ### Resource Console Integration @@ -295,9 +259,6 @@ The user profile links to the Resource Console for managing AI resources: - Finetunes (model training) - Evaluations (model testing) -Sources: -- [frontend/src/components/resource_console/ResourceConsoleIndex.vue:1-174]() -- [frontend/src/components/resource_console/EvaluationTable.vue:1-336]() ## 6. Internationalization @@ -321,11 +282,6 @@ Example of localized elements: - Validation messages - Confirmation dialogs -Sources: -- [frontend/src/locales/en_js/profile.js:1-35]() -- [frontend/src/locales/zh_js/profile.js:1-35]() -- [frontend/src/locales/en_js/navbar.js:8-9]() - Profile navigation items -- [frontend/src/locales/zh_js/navbar.js:8-9]() - Profile navigation items (Chinese) ## 7. UI Components and Data Flow @@ -361,9 +317,5 @@ The User Profile and Settings system uses Element Plus UI components and follows - After successful profile update - Maintains session with updated information -Sources: -- [frontend/src/components/user_settings/ProfileEdit.vue:31-105]() - Form fields -- [frontend/src/components/user_settings/ProfileEdit.vue:248-292]() - Data submission -- [frontend/src/components/user_settings/Profile.vue:4-14]() - Profile display The User Profile and Settings system provides a comprehensive interface for users to manage their personal information and account settings while integrating seamlessly with other components of the CSGHub platform. \ No newline at end of file diff --git a/docs/development/13_Organizations_and_Collections.md b/docs/development/13_Organizations_and_Collections.md index f396c1c78..328bace53 100644 --- a/docs/development/13_Organizations_and_Collections.md +++ b/docs/development/13_Organizations_and_Collections.md @@ -33,7 +33,7 @@ graph TD Organization -- "owns" --> Repository ``` -Sources: [frontend/src/components/organizations/OrganizationDetail.vue:6-14](), [frontend/src/components/organizations/OrganizationSettings.vue:46-53]() + ### 1.1 Organization Structure and Properties @@ -51,7 +51,7 @@ Organizations have several key properties: Organization data is fetched from the backend API endpoint `/organization/{name}` which returns the organization details. -Sources: [frontend/src/components/organizations/OrganizationDetail.vue:132-143](), [frontend/src/components/organizations/OrganizationSettings.vue:57-71]() + ### 1.2 Organization Membership and Roles @@ -83,7 +83,7 @@ graph TD ReadRole --> AccessRepos ``` -Sources: [frontend/src/components/organizations/OrganizationDetail.vue:21-68](), [frontend/src/components/organizations/OrganizationDetail.vue:156-165]() + Users can have one of the following roles in an organization: @@ -95,7 +95,7 @@ The current user's role in an organization is determined by calling the API endp Organization members are displayed in the organization detail view with their roles. The organization admin can invite new members through the `InviteMember` component. -Sources: [frontend/src/components/organizations/OrganizationDetail.vue:81-97](), [frontend/src/components/organizations/OrganizationDetail.vue:146-155]() + ### 1.3 Organization Repository Management @@ -132,7 +132,7 @@ graph TD Codes --- CodeCreate ``` -Sources: [frontend/src/components/organizations/OrganizationDetail.vue:27-63]() + Users with admin or write permissions can create repositories under the organization. The repository creation UI includes options to specify the organization as the owner. When creating a repository through the organization interface, the organization name is automatically passed as a URL parameter. @@ -161,7 +161,7 @@ graph TD Collection --- AddToCollection ``` -Sources: [frontend/src/components/collections/AddToCollections.vue:97-106](), [frontend/src/components/collections/AddToCollections.vue:129-146]() + ### 2.1 Collection Management @@ -172,7 +172,7 @@ The key operations for collections include: 1. **Viewing collections**: Fetching collections for a user 2. **Adding repositories to collections**: Adding one or more repositories to a collection -Sources: [frontend/src/components/collections/AddToCollections.vue:97-106]() + ### 2.2 Adding Repositories to Collections @@ -194,7 +194,7 @@ flowchart TD ShowSuccess --> CloseDialog["Close dialog"] ``` -Sources: [frontend/src/components/collections/AddToCollections.vue:108-146]() + The process of adding a repository to a collection involves: @@ -212,7 +212,7 @@ The data structure for adding a repository to a collection is: } ``` -Sources: [frontend/src/components/collections/AddToCollections.vue:130-135]() + ### 2.3 Collection Integration @@ -220,7 +220,7 @@ Collections are integrated with repository views. Users can add repositories to Collections are displayed in user profiles, allowing users to quickly access their organized repository groups. -Sources: [frontend/src/components/collections/AddToCollections.vue:1-5]() + ## 3. Integration with User Profiles @@ -249,11 +249,11 @@ graph TD OrgsEndpoint --> UserColls ``` -Sources: [frontend/src/components/user_settings/Profile.vue:43-66]() + The user profile displays organization memberships, showing organization logos with tooltips containing organization names. When a user views their own profile, the organization data is retrieved from the user store, while for other users' profiles, it's fetched from the API endpoint `/user/{username}`. -Sources: [frontend/src/components/user_settings/Profile.vue:100-114]() + ## 4. Summary diff --git a/docs/development/14_Frontend_Architecture.md b/docs/development/14_Frontend_Architecture.md index 3db9754c5..c2c81cd3c 100644 --- a/docs/development/14_Frontend_Architecture.md +++ b/docs/development/14_Frontend_Architecture.md @@ -39,7 +39,7 @@ graph TB end ``` -Sources: [frontend/src/main.js:1-167](), [frontend/src/admin.js:1-70](), [frontend/package.json:1-55]() + ## Core Technologies @@ -54,7 +54,7 @@ The frontend application is built using the following key technologies: | i18n | Internationalization | English and Chinese language support | | Vite | Build tool | Development and production builds | -Sources: [frontend/package.json:14-42](), [frontend/src/main.js:1-9]() + ## Application Architecture @@ -83,7 +83,7 @@ graph TD CreateAdminApp --> MountAdminApp["Mount Admin App (#admin)"] ``` -Sources: [frontend/src/main.js:75-167](), [frontend/src/admin.js:46-70]() + ### Component Structure @@ -118,7 +118,7 @@ graph TD AdminSystem --> AdminUserList["AdminUserList"] ``` -Sources: [frontend/src/main.js:16-69](), [frontend/src/main.js:146-158]() + ## State Management @@ -159,7 +159,7 @@ graph LR UserStore --> ActionLimited ``` -Sources: [frontend/src/stores/UserStore.js:1-95](), [frontend/src/packs/persistPinia.js:1-20]() + ### State Persistence @@ -169,7 +169,7 @@ A custom persistence plugin ensures that state is preserved across page reloads, 2. On initialization, checks if stored state is still valid (within expiration window) 3. Subscribes to store changes and updates localStorage -Sources: [frontend/src/packs/persistPinia.js:1-20]() + ## Routing and Navigation @@ -212,7 +212,7 @@ graph TD Router --> NewOrg ``` -Sources: [frontend/src/main.js:146-158]() + ## UI Component System @@ -251,7 +251,7 @@ graph TD CustomComponents --> RepoCards ``` -Sources: [frontend/src/main.js:160-162](), [frontend/src/admin.js:51-52]() + ## Internationalization (i18n) @@ -276,7 +276,7 @@ graph LR TranslatedText --> I18n ``` -Sources: [frontend/src/main.js:132-139](), [frontend/src/admin.js:29-36]() + ## Build and Development System @@ -310,7 +310,7 @@ graph TB ProdConfig --> ChunkSplitting ``` -Sources: [frontend/vite.config.js:32-118]() + ### Build Configuration @@ -326,7 +326,7 @@ The build system is configured to optimize the application differently for devel - Console logs removed - Optimized chunk splitting -Sources: [frontend/vite.config.js:32-76](), [frontend/vite.config.js:78-118]() + ## API Integration @@ -348,7 +348,7 @@ graph TD JsonResponse --> HandleResponse ``` -Sources: [frontend/src/main.js:324](), [frontend/src/navbar/Navbar.vue:372]() + ## Authentication Flow @@ -374,7 +374,7 @@ graph TB ClearCookies --> ClearUserStore["Clear UserStore"] ``` -Sources: [frontend/src/navbar/Navbar.vue:379-382](), [frontend/src/stores/UserStore.js:25-33]() + ## Responsive Design @@ -384,7 +384,7 @@ The frontend implements responsive design through Tailwind CSS and media queries 2. Mobile-specific navigation for smaller screens 3. Responsive components that adjust to container width -Sources: [frontend/src/navbar/Navbar.vue:3-255](), [frontend/package.json:51]() + ## Performance Optimizations @@ -395,7 +395,7 @@ The application includes several performance optimizations: 3. **Chunk Optimization**: Organizes code into logical chunks 4. **CSS Optimization**: Uses Tailwind CSS for efficient styling -Sources: [frontend/vite.config.js:83-101]() + ## Security Considerations @@ -406,7 +406,7 @@ The frontend implements several security best practices: 3. Protected routes requiring authentication 4. Input validation for user inputs -Sources: [frontend/src/stores/UserStore.js:51-58]() + ## Conclusion diff --git a/docs/development/15_Component_System_and_Navigation.md b/docs/development/15_Component_System_and_Navigation.md index a9966ab7f..434be672b 100644 --- a/docs/development/15_Component_System_and_Navigation.md +++ b/docs/development/15_Component_System_and_Navigation.md @@ -50,7 +50,7 @@ graph TD class A,N,NM,C1,C2,C3 primary; ``` -Sources: [frontend/src/main.js:1-167](), [frontend/src/components/navbar/Navbar.vue:1-443]() + ### 1.2 Component Registration and Initialization @@ -77,7 +77,7 @@ sequenceDiagram main->>app: Mount to #app element ``` -Sources: [frontend/src/main.js:75-167]() + ## 2. Navigation System @@ -119,7 +119,7 @@ graph TD class N,NM,US primary; ``` -Sources: [frontend/src/components/navbar/Navbar.vue:1-443](), [frontend/src/components/navbar/MenuItems.vue:1-236]() + ### 2.2 Menu Structure and Generation @@ -150,7 +150,7 @@ graph LR class MI,RS,SW primary; ``` -Sources: [frontend/src/components/navbar/MenuItems.vue:128-211]() + ## 3. User State Integration with Navigation @@ -205,7 +205,7 @@ graph TD class US,UA1,UR1,UI1 primary; ``` -Sources: [frontend/src/stores/UserStore.js:1-95]() + ### 3.2 Navigation State Flow @@ -257,7 +257,7 @@ sequenceDiagram end ``` -Sources: [frontend/src/components/navbar/Navbar.vue:60-315](), [frontend/src/stores/UserStore.js:25-33]() + ## 4. Responsive Design and Styling @@ -276,7 +276,7 @@ The application uses Tailwind CSS with custom breakpoints to manage responsive l | md | < 768px | Tablet portrait | | sm | < 640px | Mobile | -Sources: [frontend/tailwind.config.js:115-122]() + ### 4.2 Navigation Layout Changes @@ -310,7 +310,7 @@ graph TD class RS,SW,M primary; ``` -Sources: [frontend/src/components/navbar/MenuItems.vue:172-210](), [frontend/src/components/navbar/Navbar.vue:258-285]() + ## 5. Styling System and UI Components @@ -353,7 +353,7 @@ graph TD class V,BC,MC primary; ``` -Sources: [frontend/src/assets/stylesheets/element-plus/_variables.css:1-94](), [frontend/src/assets/stylesheets/element-plus/button.css:1-449](), [frontend/src/assets/stylesheets/element-plus/menu.css:1-36]() + ### 5.2 Responsive Width Utility @@ -371,7 +371,7 @@ This class is used in the navigation component to ensure proper width at differe
``` -Sources: [frontend/src/style.css:135-137](), [frontend/src/components/navbar/Navbar.vue:4-5]() + ## 6. State Persistence @@ -407,7 +407,7 @@ graph TD class P,PP,TC primary; ``` -Sources: [frontend/src/packs/persistPinia.js:1-20](), [frontend/src/stores/UserStore.js:89-93]() + ## 7. Admin Navigation Integration @@ -454,7 +454,7 @@ graph TD class A,AP,AN primary; ``` -Sources: [frontend/src/admin.js:1-71](), [frontend/src/main.js:146-152]() + ## 8. Component Reusability @@ -475,7 +475,7 @@ app.component('CsgButton', CsgButton) app.component('FlashMessage', FlashMessage) ``` -Sources: [frontend/src/main.js:160-162]() + ### 8.2 Element Plus Integration @@ -488,4 +488,3 @@ The navigation system extensively uses Element Plus components with custom styli These components are customized using CSS variables and custom styles to match the application's design language. -Sources: [frontend/src/components/navbar/Navbar.vue:19-39](), [frontend/src/components/navbar/Navbar.vue:44-57](), [frontend/src/components/navbar/Navbar.vue:258-285]() \ No newline at end of file diff --git a/docs/development/16_Internationalization_i18n.md b/docs/development/16_Internationalization_i18n.md index 96254b9d7..534bddb11 100644 --- a/docs/development/16_Internationalization_i18n.md +++ b/docs/development/16_Internationalization_i18n.md @@ -27,7 +27,7 @@ flowchart TD Components --- TransMethod["$t() Translation Method"] ``` -Sources: [frontend/src/locales/en_js/all.js](), [frontend/src/locales/zh_js/all.js](), [frontend/src/locales/en_js/endpoints.js](), [frontend/src/locales/zh_js/endpoints.js]() + ## Translation File Structure @@ -60,7 +60,7 @@ graph TD I18nConfig --> ZHEndpoints ``` -Sources: [frontend/src/locales/en_js/all.js](), [frontend/src/locales/zh_js/all.js](), [frontend/src/locales/en_js/endpoints.js](), [frontend/src/locales/zh_js/endpoints.js]() + ### Translation Files Content @@ -103,7 +103,7 @@ export const endpoints = { } ``` -Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/zh_js/all.js:1-118](), [frontend/src/locales/en_js/endpoints.js:1-90](), [frontend/src/locales/zh_js/endpoints.js:1-90]() + ## Using Translations in Components @@ -148,7 +148,7 @@ Here's how translations are used in actual components: ElMessage.warning(error.value.msg || t('all.fetchError')) ``` -Sources: [frontend/src/components/endpoints/EndpointPage.vue:6-7](), [frontend/src/components/datasets/ParquetViewer.vue:10-11](), [frontend/src/components/datasets/ParquetViewer.vue:44-45]() + ## Translation Key Organization @@ -175,7 +175,7 @@ graph TD end ``` -Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/en_js/endpoints.js:1-90](), [frontend/src/components/endpoints/EndpointPage.vue:1-142]() + ## Adding New Translations @@ -196,7 +196,7 @@ The system uses consistent naming patterns for translation keys: | `module.section.item` | UI section items | `endpoints.detail.endpointUrl` | | `module.section.action` | Section-specific actions | `endpoints.settings.changeVisibility` | -Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/en_js/endpoints.js:1-90]() + ## Implementation in Components @@ -240,7 +240,7 @@ const { t } = useI18n() ElMessage.warning(error.value.msg || t('all.fetchError')) ``` -Sources: [frontend/src/components/shared/RepoSummary.vue:18-19](), [frontend/src/components/shared/RepoSummary.vue:121](), [frontend/src/components/datasets/ParquetViewer.vue:236]() + ### Dynamic Content with i18n @@ -250,7 +250,7 @@ Some components like `ParquetViewer.vue` use i18n with dynamic data, combining t

{{ $t('all.subset') }}（{{ numSubsets }}）

``` -Sources: [frontend/src/components/datasets/ParquetViewer.vue:44]() + ### Loading Indicators with i18n @@ -262,7 +262,7 @@ Internationalized loading messages are used in components:
``` -Sources: [frontend/src/components/endpoints/EndpointPage.vue:71-73]() + ## Testing i18n Implementation @@ -280,7 +280,7 @@ describe("EndpointSettings", () => { When testing components that use i18n, the testing framework must include the i18n plugin setup to properly render translated content. -Sources: [frontend/src/components/__tests__/endpoints/EndpointSettings.spec.js:109-114]() + ## Conclusion diff --git a/docs/development/17_API_Integration_and_Data_Fetching.md b/docs/development/17_API_Integration_and_Data_Fetching.md index fba01fa0b..4dcad19e7 100644 --- a/docs/development/17_API_Integration_and_Data_Fetching.md +++ b/docs/development/17_API_Integration_and_Data_Fetching.md @@ -44,9 +44,6 @@ flowchart TD GinRouter --> |"routes to"| RuntimeFrameworks ``` -Sources: -- [frontend/src/packs/useFetchApi.js:1-90]() -- [frontend/src/packs/auth.js:1-41]() ### The useFetchApi Utility @@ -79,8 +76,6 @@ sequenceDiagram end ``` -Sources: -- [frontend/src/packs/useFetchApi.js:32-76]() ## Common Data Fetching Patterns @@ -100,9 +95,6 @@ const fetchResources = async () => { } ``` -Sources: -- [frontend/src/components/shared/deploy_instance/fetchResourceInCategory.js:1-53]() -- [frontend/src/components/endpoints/NewEndpoint.vue:397-408]() ### Request-Response Structure @@ -125,9 +117,6 @@ flowchart TD H --> J ``` -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:551-566]() -- [frontend/src/packs/useFetchApi.js:64-77]() ## Component API Integration Examples @@ -161,10 +150,6 @@ flowchart TD D --> A ``` -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:511-567]() -- [frontend/src/components/finetune/NewFinetune.vue:333-367]() -- [frontend/src/components/evaluations/NewEvaluation.vue:453-488]() ### Managing Resources (Settings/Updates) @@ -184,9 +169,6 @@ Below is an example from the EndpointSettings component: | Update Endpoint | `/models/${modelId}/run/${endpointId}` | PUT | Modify endpoint settings | | Delete Endpoint | `/models/${modelId}/run/${endpointId}` | DELETE | Remove an endpoint | -Sources: -- [frontend/src/components/endpoints/EndpointSettings.vue:397-563]() -- [frontend/src/components/finetune/FinetuneSettings.vue:284-379]() ## The API Request Lifecycle @@ -228,9 +210,6 @@ sequenceDiagram end ``` -Sources: -- [frontend/src/packs/useFetchApi.js:43-77]() -- [frontend/src/components/endpoints/NewEndpoint.vue:526-566]() ## Common API Endpoints and Their Usage @@ -246,11 +225,6 @@ The CSGHub frontend interacts with several key API endpoints for different opera | Clusters | `/cluster` | List available clusters | Multiple components | | Models | `/runtime_framework/models` | Search for models | Multiple components | -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:550-551]() -- [frontend/src/components/finetune/NewFinetune.vue:353-357]() -- [frontend/src/components/evaluations/NewEvaluation.vue:472-473]() -- [frontend/src/components/shared/deploy_instance/fetchResourceInCategory.js:19-20]() ## Special Feature: Engine Arguments Management @@ -266,10 +240,6 @@ flowchart TD A -->|"includes in API request"| F["API Call with engine_args"] ``` -Sources: -- [frontend/src/components/endpoints/EngineArgs.vue:1-127]() -- [frontend/src/packs/useEngineArgs.js:1-55]() -- [frontend/src/components/endpoints/NewEndpoint.vue:542-544]() This specialized system provides appropriate input controls based on parameter type and tracks changes to include only modified parameters in API requests. @@ -286,10 +256,6 @@ For authentication, the system automatically: - Handles token expiration - Manages user logout -Sources: -- [frontend/src/packs/useFetchApi.js:13-30]() -- [frontend/src/packs/useFetchApi.js:64-77]() -- [frontend/src/packs/auth.js:28-38]() ## Summary diff --git a/docs/development/18_Admin_Panel.md b/docs/development/18_Admin_Panel.md index 3b75884ac..cbaf7fe2b 100644 --- a/docs/development/18_Admin_Panel.md +++ b/docs/development/18_Admin_Panel.md @@ -23,9 +23,6 @@ flowchart TD The admin privileges check is performed using computed properties in the UserStore that check for the presence of admin roles: -Sources: -- [frontend/src/components/navbar/Navbar.vue:137-145]() -- [frontend/src/stores/UserStore.js:26-27]() ## Architecture @@ -62,9 +59,6 @@ graph TD ApiCalls --> BackendAPI["Backend Admin API"] ``` -Sources: -- [frontend/src/admin.js:46-70]() -- [frontend/src/components/admin_next/router/index.js:51-202]() ## Navigation Structure @@ -104,9 +98,6 @@ graph TD Broadcasts --> BroadcastEdit["AdminBroadcastEdit (/:id/edit)"] ``` -Sources: -- [frontend/src/components/admin_next/router/index.js:51-202]() -- [frontend/src/components/admin_next/router/index.js:34-39]() ## Features and Components @@ -116,9 +107,6 @@ The Admin Panel provides several administrative features, each implemented as a The dashboard displays an overview of the platform and administrative information. It's the default landing page for the Admin Panel. -Sources: -- [frontend/src/components/admin_next/router/index.js:51-60]() -- [frontend/src/main.js:64]() ### User Management @@ -129,9 +117,6 @@ Allows administrators to view, edit, and manage user accounts. It includes compo | AdminUserList | Displays all users with search and filtering capabilities | | AdminUserDetail | Shows detailed information for a single user | -Sources: -- [frontend/src/components/admin_next/router/index.js:70-83]() -- [frontend/src/main.js:66-67]() ### System Configuration @@ -144,20 +129,11 @@ Provides settings for configuring system-wide behavior. In addition to general s | AdminBroadcastNew | Creates new broadcast messages | | AdminBroadcastEdit | Edits existing broadcast messages | -Sources: -- [frontend/src/components/admin_next/router/index.js:61-69]() -- [frontend/src/components/admin_next/router/index.js:184-201]() -- [frontend/src/main.js:68]() ### Multi-Source Sync Provides settings for synchronizing with external sources including concurrent task settings and bandwidth limit management. -Sources: -- [frontend/src/components/admin_next/router/index.js:84-91]() -- [frontend/src/main.js:69]() -- [frontend/src/locales/zh_js/admin.js:35-43]() -- [frontend/src/locales/en_js/admin.js:35-43]() ### Model Management @@ -169,10 +145,6 @@ Allows administrators to view, edit, and manage models, including model weight s | AdminModelDetail | Shows detailed information for a single model | | AdminModelEdit | Provides form for editing model properties | -Sources: -- [frontend/src/components/admin_next/router/index.js:92-111]() -- [frontend/src/locales/zh_js/admin.js:54-64]() -- [frontend/src/locales/en_js/admin.js:54-64]() ### Serverless API Management @@ -184,11 +156,6 @@ Allows administrators to manage serverless API deployments, including deployment | AdminServerlessForm | Creates/edits serverless deployments | | AdminServerlessDetail | Shows details for a single deployment | -Sources: -- [frontend/src/components/admin_next/router/index.js:112-137]() -- [frontend/src/components/admin_next/serverless/AdminServerlessList.vue:1-132]() -- [frontend/src/locales/zh_js/admin.js:65-90]() -- [frontend/src/locales/en_js/admin.js:65-90]() ### Tag Management @@ -202,11 +169,6 @@ Allows administrators to manage tags and tag categories, which are used througho | AdminTagCategoriesList | Lists tag categories | | AdminTagCategoriesForm | Creates/edits tag categories | -Sources: -- [frontend/src/components/admin_next/router/index.js:138-163]() -- [frontend/src/components/admin_next/router/index.js:164-183]() -- [frontend/src/locales/zh_js/admin.js:92-130]() -- [frontend/src/locales/en_js/admin.js:92-130]() ## Internationalization Support @@ -219,11 +181,6 @@ The Admin Panel fully supports internationalization (i18n) with English and Chin The locale selection is determined by user preference stored in cookies, with a fallback to browser language detection. -Sources: -- [frontend/src/admin.js:29-39]() -- [frontend/src/components/admin_next/router/index.js:24-31]() -- [frontend/src/locales/en_js/admin.js:1-143]() -- [frontend/src/locales/zh_js/admin.js:1-143]() ## State Management @@ -249,10 +206,6 @@ graph TD UserStoreDefinition --> LocalStorage ``` -Sources: -- [frontend/src/stores/UserStore.js:7-93]() -- [frontend/src/admin.js:38-39]() -- [frontend/src/packs/persistPinia.js:1-20]() ## Implementation Details @@ -269,9 +222,6 @@ export const ROUTERS = [ ]; ``` -Sources: -- [frontend/src/components/admin_next/router/index.js:34-46]() -- [frontend/src/components/admin_next/router/index.js:215-220]() ### Admin Panel Initialization @@ -283,8 +233,6 @@ const app = createApp(AdminLayout); app.mount("#admin"); ``` -Sources: -- [frontend/src/admin.js:46-70]() ### Component Reuse @@ -296,17 +244,11 @@ The Admin Panel uses shared components for consistent UI patterns: - SvgIcon component for icons - CsgButton component for actions -Sources: -- [frontend/src/admin.js:51-52]() -- [frontend/src/components/admin_next/serverless/AdminServerlessList.vue:86]() ## Admin Panel vs Main Application The Admin Panel is built as a separate Vue application (`admin.js`) rather than as a component within the main application (`main.js`). This separation provides better isolation and maintainability, while allowing the Admin Panel to use the same UI components and styling as the main application. -Sources: -- [frontend/src/admin.js:46-70]() -- [frontend/src/main.js:147-152]() ## Conclusion diff --git a/docs/development/19_Backend_Architecture.md b/docs/development/19_Backend_Architecture.md index 08f5cf87c..4ee4bf9cb 100644 --- a/docs/development/19_Backend_Architecture.md +++ b/docs/development/19_Backend_Architecture.md @@ -40,7 +40,7 @@ graph TD Handlers --> S3 ``` -Sources: [internal/routes/router.go:33-95]() + ## 2. Router System @@ -77,7 +77,7 @@ sequenceDiagram Router-->>App: Return Configured Gin Engine ``` -Sources: [internal/routes/router.go:33-95]() + ### 2.2 Route Organization @@ -108,7 +108,7 @@ graph TD end ``` -Sources: [internal/routes/router.go:205-230](), [internal/routes/models.go](), [internal/routes/datasets.go](), [internal/routes/codes.go](), [internal/routes/spaces.go]() + ## 3. Middleware System @@ -136,7 +136,7 @@ graph LR Handler --> Response ``` -Sources: [internal/routes/router.go:61-72]() + ### 3.2 Route-Specific Middleware @@ -154,7 +154,7 @@ flowchart LR CheckUser -- "User Not Authenticated" --> Unauthorized ``` -Sources: [internal/routes/models.go:22-23](), [internal/routes/datasets.go:16-17](), [internal/routes/codes.go:22-23](), [internal/routes/spaces.go:16-17]() + ## 4. Handler Architecture @@ -197,7 +197,7 @@ classDiagram HandlersRegistry *-- RenderHandlerRegistry ``` -Sources: [internal/routes/router.go:26-31](), [internal/handlers/render/registry.go:9-26]() + ### 4.2 Base Handler Pattern @@ -276,7 +276,7 @@ classDiagram BaseHandlerImpl <|-- SpaceHandlerImpl ``` -Sources: [internal/handlers/render/repo.go:39-163](), [internal/handlers/render/models.go](), [internal/handlers/render/datasets.go](), [internal/handlers/render/codes.go](), [internal/handlers/render/spaces.go]() + ## 5. Authentication and Session Management @@ -314,7 +314,7 @@ sequenceDiagram CSGHub-->>User: New Token Response ``` -Sources: [internal/handlers/frontend/token.go:34-59](), [pkg/server/types/jwt.go]() + ### 5.2 Authentication Middleware @@ -329,7 +329,7 @@ Table: Authentication Middleware Components | `JwtUtils` | Utility for working with JWT tokens | | `GetCurrentUser` | Extracts user information from JWT token | -Sources: [internal/routes/router.go:69](), [internal/routes/models.go:23](), [internal/handlers/frontend/token.go:35]() + ## 6. View Rendering @@ -364,7 +364,7 @@ graph TD end ``` -Sources: [internal/routes/router.go:106-203]() + ## 7. Static File Serving @@ -384,7 +384,7 @@ graph LR end ``` -Sources: [internal/routes/router.go:232-250]() + ## 8. API Integration @@ -417,7 +417,7 @@ graph TD end ``` -Sources: [internal/routes/router.go:252-265]() + ## 9. Error Handling @@ -442,7 +442,7 @@ flowchart TD Handler -- "Handler Error" --> ErrorLogging ``` -Sources: [internal/routes/router.go:61-68]() + ## Conclusion diff --git a/docs/development/1_CSGHub_Overview.md b/docs/development/1_CSGHub_Overview.md index 871742365..fcf6f0117 100644 --- a/docs/development/1_CSGHub_Overview.md +++ b/docs/development/1_CSGHub_Overview.md @@ -47,8 +47,6 @@ flowchart TB StarhubServer --> S3 ``` -Sources: [README.md:12-14](../../frontend/src/components/navbar/Navbar.vue), [go.mod:1-19](../../go.mod), [config/config.go:14-40](../../config/config.go) - ## Core Components CSGHub is built with a modular architecture consisting of several key components: @@ -121,8 +119,6 @@ flowchart TB StarhubServer --> FinetuneMgmt ``` -Sources: [frontend/src/main.js:1-167](../../frontend/src/main.js), [config/config.go:14-40](../../config/config.go), [README.md:12-18](../../README.md) - ## Repository System The repository system is the core of CSGHub, providing management of various LLM assets: @@ -243,8 +239,6 @@ flowchart TB Session --> UserStore ``` -Sources: [frontend/src/stores/UserStore.js:1-95](../../frontend/src/stores/UserStore.js), [frontend/src/components/navbar/Navbar.vue:60-222](../../frontend/src/components/navbar/Navbar.vue), [frontend/src/packs/persistPinia.js:1-20](../../frontend/src/packs/persistPinia.js) - ## Resource Management CSGHub provides specialized tools for AI model deployment, fine-tuning, and evaluation: @@ -326,8 +320,6 @@ flowchart TB EvalRun --> EvalResults ``` -Sources: [frontend/src/main.js:45-58](../../frontend/src/main.js), [docs/csghub_saas_en.md:90-178](../../docs/csghub_saas_en.md), [docs/csghub_saas_zh.md:90-178](../../docs/csghub_saas_zh.md) - ## Internationalization CSGHub provides comprehensive internationalization support for multiple languages: @@ -366,8 +358,6 @@ flowchart TB I18n --> Messages ``` -Sources: [frontend/src/main.js:132-139](../../frontend/src/main.js), [frontend/src/components/navbar/Navbar.vue:44-57](../../frontend/src/components/navbar/Navbar.vue) - ## Deployment Options CSGHub can be deployed in several ways: @@ -379,7 +369,6 @@ CSGHub can be deployed in several ways: | Helm | Kubernetes-based deployment | Production, scalable environments | | SaaS | Cloud service at opencsg.com | Immediate access without infrastructure | -Sources: [README.md:39-47](../../README.md), [docs/release_notes.md:12-15](../../docs/release_notes.md) ## Version History @@ -391,10 +380,6 @@ CSGHub follows a regular release cycle with version updates. Key milestones incl - v1.0.0 (2024.10.15): Stable multi-sync function, Docker for ARM, region support - v0.9.0 (2024.09.15): Migrated to Go, added Collections feature -For a complete history, see the [release notes](../../docs/release_notes.md). - -Sources: [docs/release_notes.md:3-31](../../docs/release_notes.md) - ## Technical Implementation CSGHub is built using modern web technologies: @@ -450,8 +435,6 @@ flowchart TB Go --> Git ``` -Sources: [frontend/package.json:14-43](../../frontend/package.json), [go.mod:5-18](../../go.mod), [frontend/src/main.js:1-8](../../frontend/src/main.js) - ## Conclusion CSGHub provides a comprehensive platform for managing LLM assets throughout their lifecycle. Its modular architecture and multiple deployment options make it suitable for a wide range of use cases, from individual research to enterprise-scale model management. The platform continues to evolve with regular updates and new features to support the growing needs of the LLM community. \ No newline at end of file diff --git a/docs/development/20_Routing_and_Middleware.md b/docs/development/20_Routing_and_Middleware.md index a56660ef1..ffee0a070 100644 --- a/docs/development/20_Routing_and_Middleware.md +++ b/docs/development/20_Routing_and_Middleware.md @@ -46,7 +46,7 @@ flowchart TB handlers --> rg6 ``` -Sources: [internal/routes/router.go:33-96]() + ## Router Initialization @@ -77,7 +77,7 @@ sequenceDiagram Router-->>App: Return configured Gin engine ``` -Sources: [internal/routes/router.go:33-96]() + ## Middleware Components @@ -103,7 +103,7 @@ flowchart LR config --> handler["Route Handler"] ``` -Sources: [internal/routes/router.go:61-73](), [internal/routes/router.go:98-104]() + ### Authentication Middleware @@ -115,7 +115,7 @@ Key functions: - Attaches authenticated user information to the request context - Handles token refreshing for extended sessions -Sources: [internal/handlers/frontend/token.go:34-59](), [internal/routes/router.go:69]() + ## Route Groups @@ -125,7 +125,7 @@ CSGHub organizes routes into distinct groups based on functionality, making the Static routes serve frontend assets such as JavaScript, CSS, and image files directly from the filesystem. -Sources: [internal/routes/router.go:232-250]() + ### API Routes @@ -137,7 +137,7 @@ API routes handle internal AJAX requests from the frontend, providing data endpo /internal_api/upload - File upload handling ``` -Sources: [internal/routes/router.go:252-265]() + ### View Routes @@ -160,7 +160,7 @@ View routes render HTML templates for different parts of the application. These | Resource console routes | Resource management console | | Admin routes | Administrative interfaces | -Sources: [internal/routes/router.go:205-230]() + ### Resolve Routes @@ -170,7 +170,7 @@ Resolve routes handle repository content resolution, mapping repository paths to /:repo_type/:namespace/:name/resolve/:branch/*path ``` -Sources: [internal/routes/router.go:259-260]() + ## Handler Registries @@ -209,7 +209,7 @@ classDiagram HandlersRegistry --o RenderHandlerRegistry : contains ``` -Sources: [internal/routes/router.go:26-31](), [internal/handlers/render/registry.go:9-26]() + ## Template Rendering @@ -222,7 +222,7 @@ The template rendering process: 4. Templates are parsed from the embedded filesystem during initialization 5. Rendered templates are served in response to view route requests -Sources: [internal/routes/router.go:106-203]() + ## Error Handling @@ -232,7 +232,7 @@ The routing system includes comprehensive error handling to provide appropriate 2. **404 Not Found**: Redirects to a dedicated not-found page for unmatched routes 3. **Authentication Errors**: Redirects unauthenticated users to appropriate error pages -Sources: [internal/routes/router.go:61-68](), [internal/routes/router.go:267-272]() + ## JWT Authentication Flow @@ -263,7 +263,7 @@ sequenceDiagram Token-->>Client: Set cookie and return token ``` -Sources: [internal/handlers/frontend/token.go:34-59](), [pkg/server/types/jwt.go:1-45]() + ## Implementation Details @@ -278,7 +278,7 @@ The router is initialized in the `Initialize` function, which sets up the Gin en 5. Sets up the HTML renderer 6. Registers all routes -Sources: [internal/routes/router.go:33-96]() + ### Global Configuration Injection @@ -301,7 +301,7 @@ func injectConfig(config types.GlobalConfig) gin.HandlerFunc { } ``` -Sources: [internal/routes/router.go:98-104](), [internal/routes/router.go:206-211]() + ## Conclusion diff --git a/docs/development/21_Repository_Handlers.md b/docs/development/21_Repository_Handlers.md index c7f84560c..c01d0515c 100644 --- a/docs/development/21_Repository_Handlers.md +++ b/docs/development/21_Repository_Handlers.md @@ -98,7 +98,7 @@ classDiagram SpaceHandler <|.. SpaceHandlerImpl ``` -Sources: [internal/handlers/render/repo.go:39-53](), [internal/handlers/render/models.go](), [internal/handlers/render/datasets.go](), [internal/handlers/render/codes.go](), [internal/handlers/render/spaces.go]() + ## Repository Types and Handler Implementation @@ -113,7 +113,7 @@ CSGHub supports four primary repository types, each with its dedicated handler i Each specific handler implementation primarily differs in the `resourceType` field value, which is used to determine which templates to render and what type-specific data to include. -Sources: [internal/handlers/render/models.go](), [internal/handlers/render/datasets.go](), [internal/handlers/render/codes.go](), [internal/handlers/render/spaces.go]() + ## Route to Handler Mapping @@ -144,7 +144,7 @@ flowchart LR end ``` -Sources: [internal/routes/models.go](), [internal/routes/datasets.go](), [internal/routes/codes.go](), [internal/routes/spaces.go]() + ## Handler Method Behavior @@ -168,7 +168,7 @@ The `BaseHandlerImpl` provides a default implementation for all handler methods, Most methods call a common `renderShow` helper method that prepares a data context and renders the appropriate template. -Sources: [internal/handlers/render/repo.go:68-123]() + ## Request Processing Flow @@ -204,7 +204,7 @@ sequenceDiagram RenderBase-->>Client: HTML Response ``` -Sources: [internal/routes/models.go](), [internal/handlers/render/repo.go:125-163]() + ## Template Rendering and Data Context @@ -241,7 +241,7 @@ The `renderShow` method plays a key role in preparing the data context for templ RenderBaseInstance.RenderTemplate(ctx, b.resourceType+"_show", data) ``` -Sources: [internal/handlers/render/repo.go:125-143](), [internal/handlers/render/repo.go:145-163]() + ## License Handling @@ -258,7 +258,7 @@ var DefaultLicenses = [][]string{ This licensing information is used when creating new repositories and is passed to the template rendering system as a JSON string. -Sources: [internal/handlers/render/repo.go:12-37](), [internal/handlers/render/repo.go:122]() + ## Extension Points @@ -268,7 +268,7 @@ While the Repository Handlers system follows a common pattern, it includes sever 2. **Method Overriding**: Specific handlers can override any base method to provide custom behavior 3. **Template Mapping**: The template naming convention (`{resourceType}_show`) allows for resource-specific templates -Sources: [internal/handlers/render/models.go:11-17](), [internal/handlers/render/datasets.go:11-17](), [internal/handlers/render/codes.go:11-17](), [internal/handlers/render/spaces.go:11-17]() + ## Repository Handler Registration @@ -293,7 +293,7 @@ flowchart TD SpaceRoutes --> |"handlers.RenderHandler.SpaceHandler"| SpaceHandler ``` -Sources: [internal/routes/models.go:8-10](), [internal/routes/datasets.go:8-10](), [internal/routes/codes.go:8-10](), [internal/routes/spaces.go:8-10]() + ## Summary diff --git a/docs/development/22_Development_and_Deployment.md b/docs/development/22_Development_and_Deployment.md index f06cdc13a..9eb380a00 100644 --- a/docs/development/22_Development_and_Deployment.md +++ b/docs/development/22_Development_and_Deployment.md @@ -26,13 +26,13 @@ yarn build # Creates a production build yarn preview # Previews the production build locally ``` -Sources: [frontend/package.json:6-13]() + ### Backend Development The backend is written in Go using the Gin web framework. The main entry point is `cmd/csghub-portal`. The Go dependencies are managed using Go modules. -Sources: [go.mod:1-3](), [Dockerfile:14]() + ## Build Configuration @@ -64,7 +64,7 @@ The development build focuses on developer experience with: - Watch mode for immediate feedback - Manual chunk strategy for common libraries -Sources: [frontend/vite.config.js:32-76]() + #### Production Build Config @@ -89,7 +89,7 @@ The production build optimizes for end-user performance: - Clean output directory - Optimized chunk splitting for better loading performance -Sources: [frontend/vite.config.js:78-118]() + ## Build Process @@ -121,7 +121,7 @@ Vite is configured to split the code into logical chunks: - Component chunks for reusable UI components - Specialized chunks for large libraries (lodash, element-plus, highlight.js) -Sources: [frontend/vite.config.js:84-101]() + ### Backend Build Process @@ -145,7 +145,7 @@ The Go build process includes: 2. Compilation with platform-specific settings 3. Generation of the `csghub-portal` binary -Sources: [Dockerfile:9-14]() + ## Docker Build Process @@ -174,7 +174,7 @@ The Dockerfile uses: 2. A minimal final stage with only the compiled binary 3. Platform-specific build flags for cross-architecture support -Sources: [Dockerfile:1-19]() + ## Testing @@ -185,7 +185,7 @@ The frontend uses Vitest for unit testing with the following commands: - `yarn test`: Run all tests in watch mode - `yarn coverage`: Generate test coverage reports -Sources: [frontend/package.json:9-10]() + ### Backend Testing @@ -228,7 +228,7 @@ The pipeline is triggered by: - Pushes to the `csghub-staging` branch for staging deployments - Tags matching the pattern `v*.*.*` for production releases -Sources: [.gitlab-ci.yml:9-12](), [.gitlab-ci.yml:15-21]() + ### Build Stage @@ -244,7 +244,7 @@ Key configuration: - Image tag based on branch/tag name - No default attestations -Sources: [.gitlab-ci.yml:23-44]() + ### Deploy Stage @@ -256,7 +256,7 @@ The deployment process: 4. Updates the running container using docker-compose 5. Cleans up unused images -Sources: [.gitlab-ci.yml:45-69]() + ## Configuration @@ -271,7 +271,7 @@ Key configuration categories: | Database | `CSGHUB_PORTAL_DATABASE_DSN`, `CSGHUB_PORTAL_DATABASE_DIALECT` | Database connection settings | | S3 Storage | `CSGHUB_PORTAL_S3_ENDPOINT`, `CSGHUB_PORTAL_S3_ACCESS_KEY_ID` | Object storage configuration | -Sources: [config/config.go:14-41]() + ### Configuration Loading @@ -285,7 +285,7 @@ flowchart LR config --> app["Application"] ``` -Sources: [config/config.go:43-50]() + ## Troubleshooting @@ -305,7 +305,7 @@ When reporting bugs, please use the bug report template and include: - Deployment method (docker compose, helm chart) - Steps to reproduce -Sources: [.github/ISSUE_TEMPLATE/bug_report.md:12-36]() + ## Feature Requests @@ -314,4 +314,3 @@ Feature enhancement requests should: - Explain the benefits of the feature - Include links to relevant documentation or design documents -Sources: [.github/ISSUE_TEMPLATE/feature_request.md:12-21]() \ No newline at end of file diff --git a/docs/development/2_Repository_Management_System.md b/docs/development/2_Repository_Management_System.md index 855604ba7..a26ffe28b 100644 --- a/docs/development/2_Repository_Management_System.md +++ b/docs/development/2_Repository_Management_System.md @@ -45,15 +45,6 @@ graph TD end ``` -Sources: -- [frontend/src/components/shared/RepoHeader.vue:1-368]() -- [frontend/src/components/shared/RepoCards.vue:1-332]() -- [frontend/src/stores/RepoDetailStore.js:1-245]() -- [internal/routes/models.go:1-31]() -- [internal/routes/datasets.go:1-31]() -- [internal/routes/codes.go:1-31]() -- [internal/routes/spaces.go:1-32]() - ## Repository Data Model All repository types share a common data model while having type-specific extensions. The state for repositories is centrally managed through the `RepoDetailStore`. @@ -86,8 +77,6 @@ classDiagram The `RepoDetailStore` contains all properties for the currently viewed repository. It includes common properties shared by all repository types as well as type-specific properties that only apply to certain repository types. -Sources: -- [frontend/src/stores/RepoDetailStore.js:6-243]() ### Type-Specific Extensions @@ -130,12 +119,6 @@ theme: "#F5F3FF" repositories: [] ``` -Sources: -- [frontend/src/stores/RepoDetailStore.js:9-236]() -- [frontend/src/components/finetune/FinetuneDetail.vue:5-16]() -- [frontend/src/components/endpoints/EndpointDetail.vue:4-16]() -- [frontend/src/components/collections/CollectionsDetail.vue:6-17]() - ## User Interface Components The Repository Management System includes several UI components that collectively provide the repository interface. @@ -171,9 +154,6 @@ The header adapts its display based on repository type: - For finetunes: Shows finetune status and model information - For other types: Shows repository type icon and metadata -Sources: -- [frontend/src/components/shared/RepoHeader.vue:1-368]() -- [frontend/src/components/__tests__/shared/RepoHeader.spec.js:1-332]() ### Repository Cards Browser @@ -204,9 +184,6 @@ graph TD - Source (for on-premise installations) - SDK type (for spaces) -Sources: -- [frontend/src/components/shared/RepoCards.vue:1-332]() - ### Repository Detail Views Each repository type has a specialized detail view that extends a common pattern: @@ -235,11 +212,6 @@ The detail views follow a consistent pattern but adapt to each repository type: - Finetune detail: Shows training status, settings - Collection detail: Shows contained repositories -Sources: -- [frontend/src/components/finetune/FinetuneDetail.vue:1-325]() -- [frontend/src/components/endpoints/EndpointDetail.vue:1-225]() -- [frontend/src/components/collections/CollectionsDetail.vue:1-173]() - ## Repository Operations The Repository Management System supports various operations on repositories. @@ -274,10 +246,6 @@ Key filtering options include: - SDK filtering via `sdk` parameter (for spaces) - Pagination via `page` and `per` parameters -Sources: -- [frontend/src/components/shared/RepoCards.vue:254-308]() -- [frontend/src/locales/en_js/models.js:1-75]() -- [frontend/src/locales/zh_js/models.js:1-72]() ### Repository Detail Display @@ -306,12 +274,6 @@ sequenceDiagram Browser->>User: Show updated like count ``` -Sources: -- [frontend/src/components/shared/RepoHeader.vue:307-335]() -- [frontend/src/components/finetune/FinetuneDetail.vue:195-212]() -- [frontend/src/components/endpoints/EndpointDetail.vue:129-144]() -- [frontend/src/components/collections/CollectionsDetail.vue:153-162]() - ### Repository Empty States The system also handles empty states, such as when no models are available: @@ -329,9 +291,6 @@ graph TD The empty state provides guidance to users on how to add their first repositories. -Sources: -- [frontend/src/components/shared/RepoCards.vue:113-115]() -- [frontend/src/components/models/EmptyModels.vue:1-15]() ## Backend Integration @@ -365,12 +324,6 @@ The backend implements consistent route patterns across repository types: - Files route: `GET /{repoType}s/:namespace/:repo_name/files/:branch/*path` - Settings route: `GET /{repoType}s/:namespace/:repo_name/settings` -Sources: -- [internal/routes/models.go:8-31]() -- [internal/routes/datasets.go:8-31]() -- [internal/routes/codes.go:8-31]() -- [internal/routes/spaces.go:8-32]() -- [internal/handlers/render/repo.go:38-163]() ### Base Handler Implementation @@ -417,12 +370,6 @@ classDiagram This design provides consistent behaviors across repository types while allowing for type-specific customizations. -Sources: -- [internal/handlers/render/repo.go:39-53]() -- [internal/handlers/render/models.go:3-17]() -- [internal/handlers/render/datasets.go:3-17]() -- [internal/handlers/render/codes.go:3-17]() -- [internal/handlers/render/spaces.go:3-17]() ## State Management @@ -456,11 +403,6 @@ The store is initialized when a user navigates to a repository detail page. The 2. Populates common and type-specific properties 3. Uses Pinia's persistence plugin to cache data -Sources: -- [frontend/src/stores/RepoDetailStore.js:6-243]() -- [frontend/src/components/finetune/FinetuneDetail.vue:283-290]() -- [frontend/src/components/endpoints/EndpointDetail.vue:195-197]() -- [frontend/src/components/collections/CollectionsDetail.vue:167-169]() ### API Integration for State Updates @@ -482,8 +424,6 @@ sequenceDiagram This pattern ensures that UI state stays synchronized with server state. -Sources: -- [frontend/src/components/shared/RepoHeader.vue:307-335]() ## Conclusion diff --git a/docs/development/3_Repository_Detail_Components.md b/docs/development/3_Repository_Detail_Components.md index b45c4405a..92489b840 100644 --- a/docs/development/3_Repository_Detail_Components.md +++ b/docs/development/3_Repository_Detail_Components.md @@ -42,8 +42,6 @@ graph TD Collection --> RT ``` -Sources: [frontend/src/components/shared/RepoHeader.vue:1-369](), [frontend/src/stores/RepoDetailStore.js:1-246](), [frontend/src/components/shared/RepoSummary.vue:1-261]() - ## Repository Header Component The RepoHeader component is a versatile UI element that adapts its display based on the repository type. It presents critical information about a repository such as name, owner, visibility, and status. @@ -81,8 +79,6 @@ classDiagram RepoHeader --> RepoDetailStore: "accesses" ``` -Sources: [frontend/src/components/shared/RepoHeader.vue:1-180](), [frontend/src/components/shared/RepoHeader.vue:228-368]() - ### Type-Specific Headers The RepoHeader adapts to different repository types with conditional rendering: @@ -92,7 +88,6 @@ The RepoHeader adapts to different repository types with conditional rendering: 3. **Finetune Header** (lines 77-98): Displays finetune status and resource information 4. **Other Repository Header** (lines 101-157): Handles models, code, spaces with appropriate icons -Sources: [frontend/src/components/shared/RepoHeader.vue:5-157]() ### Key Features @@ -103,7 +98,6 @@ Sources: [frontend/src/components/shared/RepoHeader.vue:5-157]() - **Path Information**: Shows owner and repository path with clickable links - **Copy Functionality**: Allows copying repository path with a single click -Sources: [frontend/src/components/shared/RepoHeader.vue:26-43](), [frontend/src/components/shared/RepoHeader.vue:116-143](), [frontend/src/components/shared/RepoHeader.vue:185-196]() ## RepoDetailStore @@ -144,7 +138,6 @@ classDiagram note for RepoDetailStore "Contains specific state properties for:\n- Spaces\n- Collections\n- Finetunes\n- Endpoints" ``` -Sources: [frontend/src/stores/RepoDetailStore.js:6-244]() ### Repository Types Support @@ -155,7 +148,6 @@ The store includes specialized fields for various repository types: - **Finetune-specific**: clusterId, deployId, imageId, replicas, proxyEndpoint - **Endpoint-specific**: gitBranch, task, replica, instances, engineArgs -Sources: [frontend/src/stores/RepoDetailStore.js:10-77](), [frontend/src/stores/RepoDetailStore.js:95-158]() ### State Management Functions @@ -166,7 +158,6 @@ RepoDetailStore provides several functions to manage repository state: - `updateUserLikes(input)`: Updates user's like status - `clearStore()`: Resets the store state -Sources: [frontend/src/stores/RepoDetailStore.js:83-163]() ## Repository Summary Component @@ -207,7 +198,6 @@ graph TD RepoSummary --> ModelRelations ``` -Sources: [frontend/src/components/shared/RepoSummary.vue:1-219]() ### Key Components @@ -216,7 +206,6 @@ Sources: [frontend/src/components/shared/RepoSummary.vue:1-219]() 3. **TestEndpoint**: For model repositories with active endpoints, provides an interface for testing the model 4. **Related Repositories**: Shows relationships between the current repository and other repositories -Sources: [frontend/src/components/shared/RepoSummary.vue:5-14](), [frontend/src/components/shared/RepoSummary.vue:48-71](), [frontend/src/components/datasets/ParquetViewer.vue:1-261]() ### Data Fetching @@ -226,7 +215,6 @@ RepoSummary fetches several types of data: - Repository relations (connected repositories) - Endpoint information (for models with deployments) -Sources: [frontend/src/components/shared/RepoSummary.vue:147-196]() ## Integration with Repository Types @@ -242,7 +230,7 @@ The Repository Detail Components adapt to different repository types to provide | Finetune | Finetune icon, status display | Finetune settings | clusterId, deployId, imageId, runtimeFramework | | Collection | Collection icon, likes display | Collection settings, repositories list | avatar, theme, repositories | -Sources: [frontend/src/components/shared/RepoHeader.vue:5-157](), [frontend/src/components/finetune/FinetuneDetail.vue:1-128](), [frontend/src/components/endpoints/EndpointDetail.vue:1-63](), [frontend/src/components/collections/CollectionsDetail.vue:1-98]() + ### Repository Type Specific Components @@ -298,7 +286,7 @@ graph TD RepoDetail --> CollectionTabs ``` -Sources: [frontend/src/components/finetune/FinetuneDetail.vue:30-105](), [frontend/src/components/endpoints/EndpointDetail.vue:17-39](), [frontend/src/components/collections/CollectionsDetail.vue:30-98]() + ## Repository Status Indicators @@ -318,7 +306,7 @@ Status information is displayed in the header and is kept up-to-date using Serve - Error states show descriptive error messages - Messages are internationalized with i18n translations -Sources: [frontend/src/components/finetune/FinetuneDetail.vue:232-280](), [frontend/src/components/endpoints/EndpointDetail.vue:147-192](), [frontend/src/locales/en_js/all.js:112-117](), [frontend/src/locales/zh_js/all.js:112-117]() + ### Error States Handling @@ -331,7 +319,7 @@ OOMKilled: 'insufficient resources', CrashLoopBackOff: 'container crashes' ``` -Sources: [frontend/src/locales/en_js/all.js:112-117](), [frontend/src/components/shared/RepoHeader.vue:67-73]() + ## Internationalization Support @@ -345,7 +333,7 @@ Key localized elements include: - Visibility indicators (public/private) - Settings labels and descriptions -Sources: [frontend/src/locales/en_js/all.js:1-118](), [frontend/src/locales/zh_js/all.js:1-118]() + ## Repository Settings Integration @@ -357,7 +345,7 @@ For repositories with editable settings, the components integrate with the appro - **Endpoint/Finetune Settings**: Shows deployment configuration and allows updating settings - **Security Settings**: Controls repository visibility (public/private) -Sources: [frontend/src/components/collections/CollectionsSettings.vue:1-326]() + ## Conclusion diff --git a/docs/development/4_Repository_Clone_and_Actions.md b/docs/development/4_Repository_Clone_and_Actions.md index 6aad448ab..75c5c1772 100644 --- a/docs/development/4_Repository_Clone_and_Actions.md +++ b/docs/development/4_Repository_Clone_and_Actions.md @@ -38,7 +38,7 @@ graph TD RC --> US["UserStore"] ``` -Sources: [frontend/src/components/shared/RepoClone.vue:1-82](), [frontend/src/components/shared/RepoTabs.vue:1-17]() + ## Repository Clone Button and Dialog @@ -62,7 +62,7 @@ sequenceDiagram CD-->>U: Display cloning instructions ``` -Sources: [frontend/src/components/shared/RepoClone.vue:74-82](), [frontend/src/components/shared/RepoClone.vue:84-254]() + ### Cloning Methods @@ -84,7 +84,7 @@ pip install csghub-sdk csghub-cli download [repository-path] [-t repository-type] ``` -Sources: [frontend/src/components/shared/RepoClone.vue:102-142](), [frontend/src/components/shared/RepoClone.vue:383-411]() + #### HTTPS Tab @@ -101,7 +101,7 @@ git lfs install git clone https://[username]:[token]@[repository-url] ``` -Sources: [frontend/src/components/shared/RepoClone.vue:143-189](), [frontend/src/components/shared/RepoClone.vue:296-360]() + #### SSH Tab @@ -112,7 +112,7 @@ git lfs install git clone [ssh-clone-url] ``` -Sources: [frontend/src/components/shared/RepoClone.vue:190-223](), [frontend/src/components/shared/RepoClone.vue:303-306]() + #### SDK Tab @@ -128,7 +128,7 @@ cache_dir = '' # cache dir of download data result = snapshot_download(repo_id, cache_dir=cache_dir, endpoint=endpoint, token=token, repo_type=repo_type) ``` -Sources: [frontend/src/components/shared/RepoClone.vue:224-253](), [frontend/src/components/shared/RepoClone.vue:370-379]() + ## Repository Actions @@ -141,7 +141,7 @@ The Repository Clone and Actions system provides various action buttons dependin | Clone/Download | Downloads repository contents | All repositories with HTTP clone URL | | Add to Collections | Adds repository to user's collections | Models, datasets, codes, spaces | -Sources: [frontend/src/components/shared/RepoClone.vue:5-9](), [frontend/src/components/shared/RepoClone.vue:74-81]() + ### Model-Specific Actions @@ -178,7 +178,7 @@ flowchart TD end ``` -Sources: [frontend/src/components/shared/RepoClone.vue:22-72](), [frontend/src/components/shared/RepoClone.vue:461-519]() + ### Administrative Actions @@ -188,7 +188,7 @@ For administrators and super users, additional actions are available: |--------|-------------|------------| | Sync | Synchronizes repository from remote source | User is admin/super_user, repository source is 'opencsg', sync status is pending/inprogress/failed | -Sources: [frontend/src/components/shared/RepoClone.vue:11-20](), [frontend/src/components/shared/RepoClone.vue:326-337](), [frontend/src/components/shared/RepoClone.vue:485-503]() + ## Technical Implementation @@ -220,7 +220,7 @@ classDiagram RepoTabs --> TabContainer : includes ``` -Sources: [frontend/src/components/shared/RepoClone.vue:258-526](), [frontend/src/components/shared/RepoTabs.vue:251-384](), [frontend/src/components/shared/TabContainer.vue:74-119]() + ### Authentication Flow @@ -249,7 +249,7 @@ sequenceDiagram end ``` -Sources: [frontend/src/components/shared/RepoClone.vue:168-175](), [frontend/src/components/shared/RepoClone.vue:468-519](), [frontend/src/components/shared/RepoClone.vue:479-483]() + ### Conditional Rendering @@ -260,7 +260,7 @@ The Repository Clone and Actions system adapts its interface based on various fa - **Repository State** - Actions may be disabled based on repository status - **Feature Enablement** - Actions like deployment, fine-tuning, and evaluation can be enabled/disabled -Sources: [frontend/src/components/shared/RepoClone.vue:22-72](), [frontend/src/components/shared/RepoClone.vue:326-337]() + ## Integration with Other Systems @@ -273,7 +273,7 @@ The Repository Clone and Actions system integrates with several other systems in 5. **Evaluation** - Creates evaluation jobs 6. **Repository Detail** - Obtains repository metadata -Sources: [frontend/src/components/shared/RepoClone.vue:264-270](), [frontend/src/components/shared/RepoClone.vue:36-40](), [frontend/src/components/shared/RepoClone.vue:65-72]() + ## Summary diff --git a/docs/development/5_Repository_Browsing_and_Filtering.md b/docs/development/5_Repository_Browsing_and_Filtering.md index 25a369782..6eda9ae8f 100644 --- a/docs/development/5_Repository_Browsing_and_Filtering.md +++ b/docs/development/5_Repository_Browsing_and_Filtering.md @@ -18,7 +18,7 @@ flowchart TD RepoCards -- "Updates search params" --> UrlParams["URL Parameters"] ``` -Sources: [frontend/src/components/shared/RepoCards.vue:1-332](). [frontend/src/components/tags/TagSidebar.vue:1-188]() + ## Key Components @@ -77,7 +77,7 @@ classDiagram RepoCards --> CsgPagination : uses ``` -Sources: [frontend/src/components/shared/RepoCards.vue:1-332](). [frontend/src/components/tags/TagSidebar.vue:1-188]() + ### Repository Item Rendering @@ -94,7 +94,7 @@ models, datasets, code: grid-cols-2 xl:grid-cols-1 spaces: grid-cols-3 xl:grid-cols-2 md:grid-cols-1 ``` -Sources: [frontend/src/components/shared/RepoCards.vue:105-123](). [frontend/src/components/models/EmptyModels.vue:1-15]() + ## Filtering Capabilities @@ -123,7 +123,7 @@ const sortOptions = [ - OpenCSG - Local -Sources: [frontend/src/components/shared/RepoCards.vue:46-83](). [frontend/src/components/shared/RepoCards.vue:174-238]() + ### Tag-Based Filtering @@ -169,7 +169,7 @@ flowchart TD TagSidebar -- "Selected Tags" --> RepoCards["RepoCards.vue"] ``` -Sources: [frontend/src/components/tags/TagSidebar.vue:1-188](). [frontend/src/components/tags/TagList.vue:1-159](). [frontend/src/components/tags/frameworks/PyTorch.vue:1-52](). [frontend/src/components/tags/frameworks/TensorFlow.vue:1-26]() + ### Special Filters by Repository Type @@ -182,7 +182,7 @@ Different repository types have specialized filtering options: 2. **Spaces**: - SDK type (Gradio, Streamlit, Nginx, Docker, MCP Server) -Sources: [frontend/src/components/shared/RepoCards.vue:71-83](). [frontend/src/components/shared/RepoCards.vue:217-238]() + ## Implementation Details @@ -204,7 +204,7 @@ url = url + `&sort=${sortSelection.value}` // Additional parameters based on filters ``` -Sources: [frontend/src/components/shared/RepoCards.vue:272-308](). [frontend/src/components/shared/RepoCards.vue:311-325]() + ### Pagination @@ -222,7 +222,7 @@ The pagination component tracks: - Total item count - Emits events for page changes -Sources: [frontend/src/components/shared/RepoCards.vue:255-261](). [frontend/src/components/shared/RepoCards.vue:124-128]() + ### URL Parameter Integration @@ -244,7 +244,7 @@ const getQueryParams = () => { } ``` -Sources: [frontend/src/components/shared/RepoCards.vue:148-156](). [frontend/src/components/tags/TagSidebar.vue:94-115]() + ## Tag System Architecture @@ -283,7 +283,7 @@ flowchart TD end ``` -Sources: [frontend/src/components/tags/TagSidebar.vue:117-151](). [frontend/src/components/tags/TagList.vue:1-159]() + ## Localization Support @@ -296,7 +296,7 @@ The repository browsing system fully supports internationalization: This enables a consistent experience across languages while maintaining the same filtering capabilities. -Sources: [frontend/src/locales/en_js/models.js:1-75](). [frontend/src/locales/zh_js/models.js:1-72]() + ## Collections Integration @@ -316,7 +316,7 @@ params.append('sort', sortSelection.value) const url = `/collections?${params.toString()}` ``` -Sources: [frontend/src/components/collections/CollectionIndex.vue:1-166]() + ## Header Tags Display @@ -330,7 +330,7 @@ When browsing repositories with active tag filters, the `HeaderTags` component p This component also enables quick removal of tag filters. -Sources: [frontend/src/components/shared/HeaderTags.vue:1-244]() + ## Summary diff --git a/docs/development/6_Model_Deployment_and_Fine-tuning.md b/docs/development/6_Model_Deployment_and_Fine-tuning.md index 7a6f240a9..5d595abb7 100644 --- a/docs/development/6_Model_Deployment_and_Fine-tuning.md +++ b/docs/development/6_Model_Deployment_and_Fine-tuning.md @@ -56,11 +56,6 @@ flowchart TD Deploy --> Scaling ``` -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:1-611]() -- [frontend/src/components/finetune/NewFinetune.vue:1-416]() -- [frontend/src/components/evaluations/NewEvaluation.vue:1-553]() -- [frontend/src/components/endpoints/EngineArgs.vue:1-128]() ## Inference Endpoint System @@ -114,9 +109,6 @@ sequenceDiagram NewEndpoint-->>User: Redirect to endpoint detail page ``` -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:1-611]() -- [frontend/src/components/endpoints/EngineArgs.vue:1-128]() The form allows configuration of: @@ -133,9 +125,6 @@ The form allows configuration of: | Quantization | Optional model quantization | None | | Visibility | Public or private access | Public | -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:16-265]() -- [frontend/src/locales/en_js/endpoints.js:1-90]() ### Managing Endpoints @@ -175,8 +164,6 @@ graph TD UpdateAPI --> RefreshEndpoint["Refresh Endpoint Data"] ``` -Sources: -- [frontend/src/components/endpoints/EndpointSettings.vue:1-573]() Key management operations include: @@ -187,8 +174,6 @@ Key management operations include: 5. **Visibility Control**: Toggle between public and private access 6. **Deletion**: Permanently remove the endpoint -Sources: -- [frontend/src/components/endpoints/EndpointSettings.vue:24-276]() ### Endpoint Lifecycle and States @@ -207,9 +192,6 @@ The endpoint goes through several states during its lifecycle: | RuntimeError | Error during operation | Restart, delete | | NoAppFile | Missing application file | Delete | -Sources: -- [frontend/src/components/endpoints/EndpointSettings.vue:334-354]() -- [frontend/src/components/endpoints/EndpointPage.vue:1-142]() ## Fine-tuning System @@ -248,8 +230,6 @@ graph TD CreateFinetune --> Redirect["Redirect to finetune detail page"] ``` -Sources: -- [frontend/src/components/finetune/NewFinetune.vue:1-416]() The fine-tuning form allows configuration of: @@ -261,8 +241,6 @@ The fine-tuning form allows configuration of: | Cloud Resource | Hardware resource (GPU required) | First available | | Runtime Framework | Fine-tuning framework | Model-dependent | -Sources: -- [frontend/src/components/finetune/NewFinetune.vue:16-164]() ### Managing Fine-tuning Jobs @@ -274,8 +252,6 @@ The fine-tuning settings interface provides controls similar to endpoint managem Fine-tuning jobs share the same lifecycle states as endpoints (Building, Running, Stopped, etc.). -Sources: -- [frontend/src/components/finetune/FinetuneSettings.vue:1-387]() ## Model Evaluation System @@ -324,8 +300,6 @@ graph TD SubmitEvaluation --> Redirect["Redirect to resource console"] ``` -Sources: -- [frontend/src/components/evaluations/NewEvaluation.vue:1-553]() The evaluation form allows configuration of: @@ -340,8 +314,6 @@ The evaluation form allows configuration of: | Cloud Resource | Hardware (for dedicated resources) | Conditional | | Evaluation Framework | Framework for running evaluations | Yes | -Sources: -- [frontend/src/components/evaluations/NewEvaluation.vue:16-244]() ## Shared Components and Infrastructure @@ -378,9 +350,6 @@ graph TD useEngineArgs["getInputTypeForEngineArg()"] --> SelectInput & SwitchInput & TextInput ``` -Sources: -- [frontend/src/components/endpoints/EngineArgs.vue:1-128]() -- [frontend/src/packs/useEngineArgs.js:1-55]() The component intelligently selects input types based on parameter names and values: @@ -390,8 +359,6 @@ The component intelligently selects input types based on parameter names and val | Toggle Switch | Boolean parameters | `enable-prefix-caching`, `enforce-eager` | | Text Input | All other parameters | Custom values, numeric settings | -Sources: -- [frontend/src/packs/useEngineArgs.js:1-55]() ### Resource and Cluster Management @@ -428,12 +395,7 @@ graph TD FilterResources --> CPU & GPU & Premium DeployType & FinetuneType & EvalType --> FilterResources -``` -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:397-408]() -- [frontend/src/components/finetune/NewFinetune.vue:257-269]() -- [frontend/src/components/evaluations/NewEvaluation.vue:380-390]() Key aspects of resource management: @@ -442,8 +404,6 @@ Key aspects of resource management: 3. **Framework Compatibility**: Filtering frameworks based on selected resources 4. **Operation Types**: Different operations (deployment, fine-tuning, evaluation) filter for appropriate resources -Sources: -- [frontend/src/components/shared/deploy_instance/fetchResourceInCategory.js]() (referenced but not shown in provided files) ## Integration with Repository System @@ -453,9 +413,6 @@ The deployment and fine-tuning systems integrate with the repository management 2. **Dataset Selection**: Evaluations and fine-tuning use datasets from the repository 3. **Visibility Control**: Aligns with repository visibility settings (public/private) -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:246-252]() -- [frontend/src/components/endpoints/EndpointSettings.vue:185-224]() ## User Interface and Localization @@ -485,9 +442,6 @@ graph TD I18n --> NewEvaluation ``` -Sources: -- [frontend/src/locales/en_js/endpoints.js:1-90]() -- [frontend/src/locales/zh_js/endpoints.js:1-90]() ## API Integration @@ -507,12 +461,6 @@ All model operations interact with backend API endpoints: | Delete Fine-tune | `/models/{modelId}/finetune/{finetuneId}` | DELETE | | Create Evaluation | `/evaluations` | POST | -Sources: -- [frontend/src/components/endpoints/NewEndpoint.vue:550-553]() -- [frontend/src/components/endpoints/EndpointSettings.vue:398-400]() -- [frontend/src/components/finetune/NewFinetune.vue:353-359]() -- [frontend/src/components/finetune/FinetuneSettings.vue:285-288]() -- [frontend/src/components/evaluations/NewEvaluation.vue:472-473]() ## Summary diff --git a/docs/development/7_Creating_and_Managing_Endpoints.md b/docs/development/7_Creating_and_Managing_Endpoints.md index c513516b7..0721d1551 100644 --- a/docs/development/7_Creating_and_Managing_Endpoints.md +++ b/docs/development/7_Creating_and_Managing_Endpoints.md @@ -33,7 +33,7 @@ flowchart TD class Building,Deploying,Running,Stopped,BuildingFailed,DeployFailed,RuntimeError states; ``` -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:334-346](), [frontend/src/components/endpoints/EndpointPage.vue:64-73]() + ## 2. Creating an Endpoint @@ -67,7 +67,7 @@ sequenceDiagram UI->>User: Redirect to endpoint detail page ``` -Sources: [frontend/src/components/endpoints/NewEndpoint.vue:526-566]() + ### 2.2 Required Parameters @@ -85,7 +85,7 @@ When creating an endpoint, you need to specify the following parameters: | Visibility | Public or private access | Yes | | Quantization | Optional model quantization for optimization | No | -Sources: [frontend/src/components/endpoints/NewEndpoint.vue:25-243]() + ### 2.3 Endpoint Creation Form @@ -109,7 +109,7 @@ To create a new endpoint, follow these steps: 10. Choose visibility (public or private). 11. Click "Create" to deploy the endpoint. -Sources: [frontend/src/components/endpoints/NewEndpoint.vue:1-266](), [frontend/src/components/endpoints/NewEndpoint.vue:306-345]() + ### 2.4 Engine Arguments @@ -138,7 +138,7 @@ flowchart TD end ``` -Sources: [frontend/src/components/endpoints/EngineArgs.vue:1-127](), [frontend/src/packs/useEngineArgs.js:1-54]() + ## 3. Managing Endpoints @@ -160,7 +160,7 @@ Endpoints can be in one of several states: | DeployFailed | The endpoint failed during deployment | | RuntimeError | The endpoint encountered a runtime error | -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:334-345]() + ### 3.2 Controlling Endpoint State @@ -171,7 +171,7 @@ You can control the state of your endpoint using the following actions: These actions can be performed from the endpoint settings page. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:4-41]() + ### 3.3 Updating Endpoint Configuration @@ -199,7 +199,7 @@ flowchart LR Confirmation --> API ``` -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:44-276]() + #### 3.3.1 Updating Cloud Resources @@ -210,7 +210,7 @@ You can change the hardware resources allocated to your endpoint. This is useful 3. Select a new cloud resource from the dropdown. 4. The change will be applied when you restart the endpoint. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:44-81]() + #### 3.3.2 Updating Runtime Framework @@ -221,7 +221,7 @@ You can change the runtime framework used by your endpoint: 3. Select a new compatible runtime framework from the dropdown. 4. The change will be applied when you restart the endpoint. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:84-124]() + #### 3.3.3 Updating Engine Arguments @@ -234,7 +234,7 @@ You can modify engine arguments to optimize performance: 5. Click "Update" to save changes. 6. The changes will be applied when you restart the endpoint. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:111-124]() + #### 3.3.4 Updating Replica Count @@ -245,7 +245,7 @@ You can adjust the minimum and maximum number of replicas for your endpoint: 3. Select new values for minimum and maximum replicas. 4. The changes will be applied when you restart the endpoint. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:128-182]() + #### 3.3.5 Changing Visibility @@ -257,7 +257,7 @@ You can change the visibility of your endpoint between public and private: 4. Confirm the change in the confirmation dialog. 5. The new visibility will be applied when you restart the endpoint. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:186-224]() + ### 3.4 Deleting an Endpoint @@ -271,7 +271,7 @@ To delete an endpoint: **Note**: Deletion is irreversible and will permanently remove the endpoint and all its files. -Sources: [frontend/src/components/endpoints/EndpointSettings.vue:227-276]() + ## 4. Viewing Endpoint Information @@ -302,7 +302,7 @@ flowchart TD EndpointPlayground --> ImageTextToText["Image Text to Text"] ``` -Sources: [frontend/src/components/endpoints/EndpointPage.vue:1-73]() + ## 5. Using the Endpoint Playground @@ -315,7 +315,7 @@ The playground provides a user-friendly interface to: 3. Adjust parameters for inference 4. Test different inputs -Sources: [frontend/src/components/endpoints/EndpointPage.vue:64-70](), [frontend/src/locales/en_js/endpoints.js:72-89]() + ## 6. Endpoint API Overview @@ -331,7 +331,7 @@ Key API endpoints for managing endpoints include: | `/models/{modelId}/run/{endpointId}/start` | PUT | Start or restart an endpoint | | `/models/{modelId}/run/{endpointId}` | DELETE | Delete an endpoint | -Sources: [frontend/src/components/endpoints/NewEndpoint.vue:550-552](), [frontend/src/components/endpoints/EndpointSettings.vue:398-424](), [frontend/src/components/endpoints/EndpointSettings.vue:485-518]() + ## 7. Comparison with Fine-tuning and Evaluation @@ -347,4 +347,3 @@ CSGHub provides three main types of computing resources: Endpoints, Fine-tuning, For more details on fine-tuning and evaluation, see [Model Fine-tuning](#3.2) and [Model Settings and Evaluation](#3.3). -Sources: [frontend/src/components/endpoints/NewEndpoint.vue](), [frontend/src/components/finetune/NewFinetune.vue](), [frontend/src/components/evaluations/NewEvaluation.vue]() \ No newline at end of file diff --git a/docs/development/8_Model_Fine-tuning.md b/docs/development/8_Model_Fine-tuning.md index 187f6ccd6..28c86a995 100644 --- a/docs/development/8_Model_Fine-tuning.md +++ b/docs/development/8_Model_Fine-tuning.md @@ -52,7 +52,7 @@ flowchart TD class NewFinetune,FinetuneSettings,EngineArgs component; ``` -Sources: [frontend/src/components/finetune/NewFinetune.vue:1-375](), [frontend/src/components/finetune/FinetuneSettings.vue:1-386](), [frontend/src/components/endpoints/EngineArgs.vue:1-127]() + ### Fine-tuning Creation Process @@ -83,7 +83,7 @@ sequenceDiagram NewFinetune->>User: Redirect to finetune detail page ``` -Sources: [frontend/src/components/finetune/NewFinetune.vue:168-367]() + ## 2. Creating a Fine-tuning Task @@ -100,7 +100,7 @@ To create a fine-tuning task, the following parameters must be specified: | runtime_framework_id | Framework for fine-tuning | Yes | | secure_level | Visibility setting (1=public, 2=private) | Yes | -Sources: [frontend/src/components/finetune/NewFinetune.vue:184-191]() + ### Model Selection Interface @@ -114,7 +114,7 @@ flowchart LR FrameworkAPI -->|"returns"| FrameworkOptions["Compatible Frameworks"] ``` -Sources: [frontend/src/components/finetune/NewFinetune.vue:320-331]() + ### Resource and Framework Selection @@ -126,7 +126,7 @@ Resource selection follows a hierarchical process: The system filters available frameworks based on the selected resource type to ensure compatibility. -Sources: [frontend/src/components/finetune/NewFinetune.vue:258-269](), [frontend/src/components/finetune/NewFinetune.vue:295-306]() + ## 3. Managing Fine-tuning Tasks @@ -146,7 +146,7 @@ stateDiagram-v2 Failed --> [*]: Delete task ``` -Sources: [frontend/src/components/finetune/FinetuneSettings.vue:256-276]() + ### Starting and Stopping Tasks @@ -156,13 +156,13 @@ The controls become available based on the current status of the fine-tuning tas - Stop button is only active for initialized or running tasks - Start button is only active for stopped tasks -Sources: [frontend/src/components/finetune/FinetuneSettings.vue:51-85](), [frontend/src/components/finetune/FinetuneSettings.vue:284-298]() + ### Deleting Fine-tuning Tasks To delete a fine-tuning task, users must confirm by typing the full task identifier in the format `finetuneName/finetuneId`. This prevents accidental deletion of valuable fine-tuning tasks. -Sources: [frontend/src/components/finetune/FinetuneSettings.vue:166-214](), [frontend/src/components/finetune/FinetuneSettings.vue:334-377]() + ## 4. Configuration Options @@ -176,7 +176,7 @@ Each resource includes: - Hardware type (e.g., GPU, CPU) - Region information -Sources: [frontend/src/components/finetune/NewFinetune.vue:258-269]() + ### Runtime Frameworks @@ -186,7 +186,7 @@ Runtime frameworks define the software stack used for fine-tuning. Available fra The system dynamically updates available frameworks when users change their resource selection. -Sources: [frontend/src/components/finetune/NewFinetune.vue:280-293](), [frontend/src/components/finetune/NewFinetune.vue:295-306]() + ### Engine Arguments @@ -199,7 +199,7 @@ Advanced configuration is available through engine arguments, which allow fine-g | Feature Toggles | enable-prefix-caching, enforce-eager | Switch | | Load Format | auto, pt, safetensors, etc. | Select | -Sources: [frontend/src/components/endpoints/EngineArgs.vue:1-127](), [frontend/src/packs/useEngineArgs.js:1-54]() + ## 5. API Integration Points @@ -215,7 +215,7 @@ Sources: [frontend/src/components/endpoints/EngineArgs.vue:1-127](), [frontend/s | `/runtime_framework/models?deploy_type=2` | GET | List fine-tunable models | | `/models/{modelId}/runtime_framework?deploy_type=2` | GET | List frameworks for model | -Sources: [frontend/src/components/finetune/NewFinetune.vue:353-366](), [frontend/src/components/finetune/FinetuneSettings.vue:284-298](), [frontend/src/components/finetune/FinetuneSettings.vue:334-352]() + ### Fine-tuning Request Format @@ -232,7 +232,7 @@ The following JSON structure is used when creating a new fine-tuning task: } ``` -Sources: [frontend/src/components/finetune/NewFinetune.vue:184-191](), [frontend/src/components/finetune/NewFinetune.vue:348-353]() + ## 6. Integration with Other Systems @@ -266,4 +266,3 @@ flowchart TD - **Endpoints**: Fine-tuned models can be deployed through the endpoint system - **Evaluation**: Models can be evaluated before and after fine-tuning to measure improvement -Sources: [frontend/src/components/finetune/NewFinetune.vue:363-365](), [frontend/src/components/finetune/FinetuneSettings.vue:349-351]() \ No newline at end of file diff --git a/docs/development/9_Model_Settings_and_Evaluation.md b/docs/development/9_Model_Settings_and_Evaluation.md index 7c8696e67..931bf20c9 100644 --- a/docs/development/9_Model_Settings_and_Evaluation.md +++ b/docs/development/9_Model_Settings_and_Evaluation.md @@ -39,10 +39,6 @@ graph TD The `ModelSettings.vue` component provides a comprehensive interface for managing model metadata and settings. It communicates with backend APIs to fetch and update model information. -Sources: -- [frontend/src/components/models/ModelSettings.vue:1-286](). The main model settings component. -- [frontend/src/components/models/ModelSettings.vue:289-681](). Logic implementation for model settings. - ## 2. Model Metadata Management ### 2.1 Basic Metadata Fields @@ -58,10 +54,6 @@ The ModelSettings component allows editing the following model metadata: Each field has its own update method that validates input and makes the necessary API calls to update the model metadata. -Sources: -- [frontend/src/components/models/ModelSettings.vue:27-49](). Nickname field and update functionality. -- [frontend/src/components/models/ModelSettings.vue:53-76](). Description field and update functionality. -- [frontend/src/components/models/ModelSettings.vue:202-242](). Visibility settings implementation. ### 2.2 Tag Management System @@ -121,12 +113,6 @@ The tag management system provides interfaces for both regular tags (stored in R - Updated via the updateIndustryTagsAPI() method - Directly linked to the model -Sources: -- [frontend/src/components/models/ModelSettings.vue:80-134](). Task tags implementation. -- [frontend/src/components/models/ModelSettings.vue:138-198](). Industry tags implementation. -- [frontend/src/components/models/ModelSettings.vue:539-549](). updateTags() method. -- [frontend/src/components/models/ModelSettings.vue:574-608](). updateTagsInReadme() and updateReadme() methods. -- [frontend/src/components/models/ModelSettings.vue:609-624](). updateIndustryTagsAPI() method. ### 2.3 Visibility Controls @@ -145,11 +131,6 @@ graph TD The visibility toggle interface provides a dropdown with "Public" and "Private" options. When changed, the system confirms the change with the user before updating the model visibility status. -Sources: -- [frontend/src/components/models/ModelSettings.vue:202-242](). Visibility settings UI. -- [frontend/src/components/models/ModelSettings.vue:499-532](). changeVisibility() method. -- [frontend/src/components/models/ModelSettings.vue:534-538](). changeVisibilityCall() method. -- [frontend/src/components/models/ModelSettings.vue:650-666](). updateModel() method. ### 2.4 Model Deletion @@ -168,10 +149,6 @@ graph TD This multi-step confirmation process helps prevent accidental deletion of models. -Sources: -- [frontend/src/components/models/ModelSettings.vue:244-285](). Model deletion UI. -- [frontend/src/components/models/ModelSettings.vue:401-410](). clickDelete() method. -- [frontend/src/components/models/ModelSettings.vue:484-497](). deleteModel() method. ## 3. Model Evaluation System @@ -217,11 +194,6 @@ graph TD The evaluation table provides a unified interface for viewing and managing model evaluations across different datasets. -Sources: -- [frontend/src/components/resource_console/EvaluationTable.vue:1-142](). Evaluation table UI components. -- [frontend/src/components/resource_console/EvaluationTable.vue:229-242](). fetchEvaluations() method. -- [frontend/src/components/resource_console/EvaluationTable.vue:244-253](). deleteEvaluation() method. -- [frontend/src/components/resource_console/EvaluationTable.vue:255-259](). detailEvaluation() method. ### 3.2 Evaluation Data Structure @@ -240,11 +212,6 @@ Each evaluation record contains the following key information: The table displays evaluation metadata and provides actions to download results, view details, or delete evaluations. -Sources: -- [frontend/src/components/resource_console/EvaluationTable.vue:15-42](). Evaluation name and model display. -- [frontend/src/components/resource_console/EvaluationTable.vue:44-90](). Dataset information display. -- [frontend/src/components/resource_console/EvaluationTable.vue:92-139](). Status and actions display. -- [frontend/src/components/resource_console/EvaluationTable.vue:190-224](). Status mapping and components. ### 3.3 Evaluation Status Workflow @@ -276,10 +243,6 @@ graph LR The evaluation status determines which actions are available to the user, with completed evaluations enabling result viewing and downloading. -Sources: -- [frontend/src/components/resource_console/EvaluationTable.vue:105-122](). Status component rendering. -- [frontend/src/components/resource_console/EvaluationTable.vue:148-176](). Action buttons based on status. -- [frontend/src/components/resource_console/EvaluationTable.vue:190-225](). Status mapping and component selection. ## 4. Integration with Resource Console @@ -306,9 +269,6 @@ graph TD The Resource Console integrates evaluations with other resource types (endpoints, finetunes) to provide a comprehensive management interface. -Sources: -- [frontend/src/components/resource_console/ResourceConsoleIndex.vue:47-57](). Evaluation section in Resource Console. -- [frontend/src/components/resource_console/EvaluationTable.vue:1-336](). Evaluation table component. ## 5. Related UI Components @@ -327,9 +287,6 @@ Status components provide visual indicators for different evaluation states: These components maintain a consistent visual language across the application. -Sources: -- [frontend/src/components/resource_console/EvaluationTable.vue:195-198](). Status component imports. -- [frontend/src/components/resource_console/EvaluationTable.vue:218-226](). Status component mapping. ### 5.2 Pagination Component @@ -344,9 +301,6 @@ graph LR This component ensures consistent pagination across the application and handles interactions with the backend API. -Sources: -- [frontend/src/components/resource_console/EvaluationTable.vue:180-186](). Pagination component in evaluation table. -- [frontend/src/components/resource_console/EvaluationTable.vue:229-242](). Pagination-aware fetchEvaluations method. ## 6. Backend API Integration @@ -364,12 +318,6 @@ Both model settings and evaluation components interact with backend APIs to fetc | `/models/{path}/tags/industry` | POST | Update industry tags | updateIndustryTagsAPI() | | `/tags` | GET | Fetch available tags | getIndustryTags() | -Sources: -- [frontend/src/components/models/ModelSettings.vue:484-497](). deleteModel() implementation. -- [frontend/src/components/models/ModelSettings.vue:565-573](). fetchReadme() implementation. -- [frontend/src/components/models/ModelSettings.vue:586-608](). updateReadme() implementation. -- [frontend/src/components/models/ModelSettings.vue:609-624](). updateIndustryTagsAPI() implementation. -- [frontend/src/components/models/ModelSettings.vue:650-666](). updateModel() implementation. ### 6.2 Evaluation API Endpoints @@ -379,10 +327,6 @@ Sources: | `/evaluations/{id}` | DELETE | Delete evaluation | deleteEvaluation() | | `/evaluations/{id}` | GET | View evaluation details | detailEvaluation() | -Sources: -- [frontend/src/components/resource_console/EvaluationTable.vue:229-242](). fetchEvaluations() implementation. -- [frontend/src/components/resource_console/EvaluationTable.vue:244-253](). deleteEvaluation() implementation. -- [frontend/src/components/resource_console/EvaluationTable.vue:255-259](). detailEvaluation() implementation. ## 7. Internationalization Support @@ -394,11 +338,6 @@ Both model settings and evaluation interfaces support internationalization (i18n | Evaluation Table | evaluation.list.* | Translated strings for column headers and actions | | Status Labels | Direct mapping | statusMapping object maps English to Chinese | -Sources: -- [frontend/src/components/models/ModelSettings.vue:8-12](). i18n for model name section. -- [frontend/src/components/resource_console/EvaluationTable.vue:211-216](). Status mapping for different languages. -- [frontend/src/locales/en_js/finetune.js:1-56](). English translations. -- [frontend/src/locales/zh_js/finetune.js:1-56](). Chinese translations. ## Summary From 3e55807eb6bc53cd5eb6cd36b90d3075b664608a Mon Sep 17 00:00:00 2001 From: zhendi Date: Sat, 31 May 2025 17:19:34 +0800 Subject: [PATCH 5/6] docs: add spacing in deployment documentation Added a blank line for better readability. --- docs/development/6_Model_Deployment_and_Fine-tuning.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/development/6_Model_Deployment_and_Fine-tuning.md b/docs/development/6_Model_Deployment_and_Fine-tuning.md index 5d595abb7..75a828219 100644 --- a/docs/development/6_Model_Deployment_and_Fine-tuning.md +++ b/docs/development/6_Model_Deployment_and_Fine-tuning.md @@ -397,6 +397,7 @@ graph TD DeployType & FinetuneType & EvalType --> FilterResources + Key aspects of resource management: 1. **Clusters**: Geographic regions where resources are deployed From 9837a90c7ad3b19237874a50cbb8e6455b89cedb Mon Sep 17 00:00:00 2001 From: zhendi Date: Sat, 31 May 2025 17:21:06 +0800 Subject: [PATCH 6/6] fix(docs): correct formatting in deployment guide Key aspects of resource management: - Fixed indentation issue - Improved readability of the section --- docs/development/6_Model_Deployment_and_Fine-tuning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/6_Model_Deployment_and_Fine-tuning.md b/docs/development/6_Model_Deployment_and_Fine-tuning.md index 75a828219..a53c7e867 100644 --- a/docs/development/6_Model_Deployment_and_Fine-tuning.md +++ b/docs/development/6_Model_Deployment_and_Fine-tuning.md @@ -395,7 +395,7 @@ graph TD FilterResources --> CPU & GPU & Premium DeployType & FinetuneType & EvalType --> FilterResources - +``` Key aspects of resource management: