Skip to content

databaseSnake is a fast, type-safe Python file-based search engine that lets you quickly query CSV, TXT or SQL dump files with memory-mapped I/O and parallel search.

Notifications You must be signed in to change notification settings

xsyncio/databaseSnake

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

databaseSnake

Typing SVG

Stars Issues Python

Author Type Safe Version


   ▄▄                  ▄▄                       ▄▄▄▄▄▄▄                          
   ██        ██        ██                      █████▀▀▀             ▄▄           
▄████  ▀▀█▄ ▀██▀▀ ▀▀█▄ ████▄  ▀▀█▄ ▄█▀▀▀ ▄█▀█▄  ▀████▄  ████▄  ▀▀█▄ ██ ▄█▀ ▄█▀█▄ 
██ ██ ▄█▀██  ██  ▄█▀██ ██ ██ ▄█▀██ ▀███▄ ██▄█▀    ▀████ ██ ██ ▄█▀██ ████   ██▄█▀ 
▀████ ▀█▄██  ██  ▀█▄██ ████▀ ▀█▄██ ▄▄▄█▀ ▀█▄▄▄ ███████▀ ██ ██ ▀█▄██ ██ ▀█▄ ▀█▄▄▄ 

🚀 Features

Blazing Fast

  • Memory-mapped I/O for files >10MB
  • Parallel search with ThreadPoolExecutor
  • LRU caching for repeated queries
  • Optimized casefold() for case-insensitive search

🔒 Type Perfect

  • basedpyright at strictest settings
  • Zero errors, zero warnings
  • Full type annotations everywhere
  • No # type: ignore anywhere

📁 Multi-Format Support

  • CSV files (comma-separated)
  • TXT files (line-by-line)
  • SQL files (data dumps)
  • Automatic parser selection

🎨 Beautiful Console

  • Rich console output with colors
  • Progress animations
  • Styled search results
  • TTY-safe color detection

📦 Installation

# Clone the repository
git clone https://github.com/xsyncio/databaseSnake.git
cd databaseSnake

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Linux/Mac
# .venv\Scripts\activate   # Windows

# Install dependencies
pip install -r requirements.txt
# Or install as package
pip install -e .

🐍 Quick Start

# Run the application
python main.py

# Or run as module
python -m dbsearcher
📸 Screenshot Preview
╭────────────────────────────────────────────────────────────────────────────╮
│    ▄▄                  ▄▄                       ▄▄▄▄▄▄▄                    │
│    ██        ██        ██                      █████▀▀▀             ▄▄     │
│ ▄████  ▀▀█▄ ▀██▀▀ ▀▀█▄ ████▄  ▀▀█▄ ▄█▀▀▀ ▄█▀█▄  ▀████▄  ████▄  ▀▀█▄ ██ ▄█▀ │
│ ██ ██ ▄█▀██  ██  ▄█▀██ ██ ██ ▄█▀██ ▀███▄ ██▄█▀    ▀████ ██ ██ ▄█▀██ ████  │
│ ▀████ ▀█▄██  ██  ▀█▄██ ████▀ ▀█▄██ ▄▄▄█▀ ▀█▄▄▄ ███████▀ ██ ██ ▀█▄██ ██ ▀█▄│
╰────────────────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────╮
│ v1.0.0 | by Xsyncio | https://github.com/xsyncio     │
╰──────────────────────────────────────────────────────╯

📁 Files: 18 | 💾 Size: 9718.45 MB

🔍 Search Examples:
• User ID: 556343434
• Phone: +19000000000
• Name: John
• Email: example@mail.eg

1. Search in databases
2. Exit

Choose an option: _

🏗️ Architecture

graph TD
    A[Terminal] --> B["__main__.py"]
    B --> C[databaseSnake]
    C --> D[SearchEngine]
    D --> E[FileIndexer]
    D --> F[ThreadPoolExecutor]
    D --> G{Parser Selection}
    G --> H[TextParser - .txt]
    G --> I[CSVParser - .csv]
    G --> J[SQLParser - .sql]
    C --> K[Rich Console UI]
    D --> L[mmap for large files]
Loading

📁 Project Structure

databaseSnake/
├── 📄 main.py                 # Entry point
├── 📄 pyproject.toml          # Project config
├── 📂 base/                   # Your database files go here
│   ├── data.csv
│   ├── users.txt
│   └── dump.sql
└── 📂 dbsearcher/             # Main package
    ├── 📄 __init__.py         # Package init
    ├── 📄 __main__.py         # CLI entry
    ├── 📄 constants.py        # Configuration
    ├── 📄 exceptions.py       # Custom errors
    ├── 📄 logging.py          # Rich logging
    ├── 📄 types.py            # Type definitions
    ├── 📂 search/             # Search engine
    │   ├── engine.py          # Main coordinator
    │   ├── indexer.py         # File indexer
    │   ├── parsers.py         # Format parsers
    │   └── results.py         # Result formatting
    ├── 📂 ui/                  # Console UI
    │   ├── colors.py          # ANSI colors
    │   ├── display.py         # Display utils
    │   ├── effects.py         # Animations
    │   └── menu.py            # Menu system
    └── 📂 utils/              # Utilities
        ├── external.py        # URL opening
        ├── filesystem.py      # File operations
        └── platform.py        # Platform detection

⚙️ Performance Optimizations

Technique Description Benefit
🗺️ mmap Memory-mapped file I/O Zero-copy reads for >10MB files
🧵 ThreadPool Parallel file processing N× speedup on multi-core
📦 LRU Cache Result caching Instant repeated queries
🔤 casefold() Optimized case folding Faster than .lower()
⏱️ Early Exit Max results limit Stops at first N matches
🔄 Generators Streaming iteration Zero full-list allocation

🛠️ Configuration

Edit dbsearcher/constants.py to customize:

# Performance tuning
MMAP_THRESHOLD_BYTES = 10 * 1024 * 1024  # Use mmap for files > 10MB
DEFAULT_PARALLEL_WORKERS = 4             # Number of parallel workers
MAX_RESULTS_DEFAULT = 10000              # Maximum results to return

# Supported file types
SUPPORTED_EXTENSIONS = (".csv", ".txt", ".sql")

🧪 Type Checking

This project uses basedpyright at the strictest settings possible:

# Install basedpyright
pip install basedpyright

# Run type checker
basedpyright dbsearcher/

# Expected output:
# 0 errors, 0 warnings, 0 notes ✅

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.


🙏 Credits

Xsyncio

Built with 🐍 Python and ❤️ by Xsyncio


About

databaseSnake is a fast, type-safe Python file-based search engine that lets you quickly query CSV, TXT or SQL dump files with memory-mapped I/O and parallel search.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages