Redis v 0.3.0rc0 (#1003)

* chore: init new broker feature * test: add redis tests to CI * feat: init Redis Pub/Sub * feat: add little features * test: add redis tests * refactor: use specific for redis type default polling_interval * feat: add Redis schemas * chore: fix some mypy * docs: fix typos * feat: stream subscriber * docs: add Redis section * feat: add consumer groups * feat: new Redis Streams consuming * refactor: use message_id_ln everywhere * feat: Redis Streams complete * chore: rm asyncapi-tool file * test: add Redis AsyncAPI tests * lint: fix some mypy * feat: automatic TestClient connect_only argument * lint: fix 3.8 list * chore: fix warning link * docs: add Redis examples * refactor: rename ext argument * chore: add watchfiles to test dependencies * feat: add NatsMessage.in_progress method * docs: remove useless from search * docs: add minify plugin to reduce build size ~10Mb * docs: add empty pages * remove useless deps * lint: fix AST types * lint: fix Redis type * lint: fix RMQ parser * lint: fix some errors * lint: fix some errors * lint: fix some mypy errors * lint: fix some mypy errors * fix: support all subsriber types with RedisRouter * lint: fix some mypy * lint: finalize Broker url type * lint: fix mypy * refactor: reuse parent method in messages * fix: correct CLI run * fix: do not use is_installed * fix: remove is_installed everuwhere * docs: update docs scripts * docs: update CONTRIBUTING * docs: add lifespan page * test: cover lifespan context doc * Update redix index documentation * Add docs for redis list sub and list batch sub * Add docs for redis stream sub and batch sub * Add docs for channel subscription * Add basic index content for redis channel, stream, list * Add redis channel publish documentation * Add docs for redis stream publishing * Add documentation for redis list publish * Add documentation for redis rpc * Add docs for redis message information * Add documentation for stream groups * Add ack redis placeholder * Add deleted API docs * Update version to 0.3.0rc0 * Add pydantic[email] as docs requirement * Revert adding pydantic[email] * Check email_validator is installed * fix: chech email_validator with import * test: use local tests in CLI * test: mark with tests as 3.9+ * test: mark pydantic_settings tests as pydanticV2 required * polishing * Fix broken list in kafka docs * lint: fix mypy * mypy fix * polishing * lint: add redis pyi files * Add docs for redis stream acknowledgement * Add cosumer group to StreamSub in redis docs examples * fix: correct acknowledgement * fix: correct redis stop * fix: do not interrupt consume cycle with exc * fix #1000: use default_log_context to fallback * Update deploy docs to publish rc as non latest version * docs: stream as a keyword argument * polishing --------- Co-authored-by: Lancetnik <[email protected]> Co-authored-by: Davor Runje <[email protected]> Co-authored-by: Nikita Pastukhov <[email protected]>
airtai · Nov 30, 2023 · 87ed808 · 87ed808
1 parent 9207feb
commit 87ed808
Show file tree

Hide file tree

Showing 574 changed files with 9,280 additions and 1,719 deletions.
diff --git a/.github/workflows/deploy-docs.yaml b/.github/workflows/deploy-docs.yaml
@@ -6,6 +6,7 @@ on:
     paths:
       - docs/**
       - .github/workflows/deploy-docs.yaml
+      - faststream/__about__.py
 
 permissions:
   contents: write
@@ -26,10 +27,17 @@ jobs:
       - run: pip install -e ".[dev]"
       - run: ./scripts/build-docs.sh
       - run: echo "VERSION=$(python3 -c 'from importlib.metadata import version; print(".".join(version("faststream").split(".")[:2]))')" >> $GITHUB_ENV
+      - run: echo "IS_RC=$(python3 -c 'from importlib.metadata import version; print("rc" in version("faststream"))')" >> $GITHUB_ENV
       - name: Configure Git user
         run: |
           git config --local user.email "github-actions[bot]@users.noreply.github.com"
           git config --local user.name "github-actions[bot]"
       - run: echo $VERSION
-      - run: cd docs && mike deploy -F mkdocs.yml --update-aliases $VERSION latest
-      - run: cd docs && mike set-default --push --allow-empty -F mkdocs.yml latest
+      - run: echo $IS_RC
+      - run: |
+          if [ "$IS_RC" == "False" ]; then
+            cd docs && mike deploy -F mkdocs.yml --update-aliases $VERSION latest
+            mike set-default --push --allow-empty -F mkdocs.yml latest
+          else
+            cd docs && mike deploy --push -F mkdocs.yml --update-aliases $VERSION
+          fi
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
@@ -22,7 +22,7 @@ jobs:
         run: |
           set -ux
           python -m pip install --upgrade pip
-          pip install -e ".[docs,rabbit,kafka,nats,lint]"
+          pip install -e ".[docs,rabbit,kafka,redis,nats,lint]"
       - name: Run mypy
         shell: bash
         run: mypy faststream tests/mypy
@@ -59,7 +59,7 @@ jobs:
           key: ${{ runner.os }}-python-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}-test-v03
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[rabbit,kafka,nats,docs,testing]
+        run: pip install .[rabbit,kafka,nats,redis,docs,testing]
       - name: Install Pydantic v1
         if: matrix.pydantic-version == 'pydantic-v1'
         run: pip install "pydantic>=1.10.0,<2.0.0"
@@ -68,7 +68,7 @@ jobs:
         run: pip install --pre "pydantic>=2.0.0b2,<3.0.0"
       - run: mkdir coverage
       - name: Test
-        run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not rabbit)) or (not nats and not kafka and not rabbit)"
+        run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not rabbit and not redis)) or (not nats and not kafka and not rabbit and not redis)"
         env:
           COVERAGE_FILE: coverage/.coverage.${{ runner.os }}-py${{ matrix.python-version }}
           CONTEXT: ${{ runner.os }}-py${{ matrix.python-version }}
@@ -91,9 +91,9 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[rabbit,kafka,nats,docs,testing]
+        run: pip install .[rabbit,kafka,nats,redis,docs,testing]
       - name: Test
-        run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not rabbit)) or (not nats and not kafka and not rabbit)"
+        run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not rabbit and not redis)) or (not nats and not kafka and not rabbit and not redis)"
 
   test-windows-latest:
     if: github.event.pull_request.draft == false
@@ -108,9 +108,9 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[rabbit,kafka,nats,docs,testing]
+        run: pip install .[rabbit,kafka,nats,redis,docs,testing]
       - name: Test
-        run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not rabbit)) or (not nats and not kafka and not rabbit)"
+        run: bash scripts/test.sh -m "(slow and (not nats and not kafka and not rabbit and not redis)) or (not nats and not kafka and not rabbit and not redis)"
 
   test-kafka-real:
     if: github.event.pull_request.draft == false
@@ -141,7 +141,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,rabbit,docs,testing]
+        run: pip install .[nats,kafka,rabbit,redis,docs,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and kafka) or kafka"
@@ -189,7 +189,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,rabbit,docs,testing]
+        run: pip install .[nats,kafka,rabbit,redis,docs,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and rabbit) or rabbit"
@@ -237,7 +237,7 @@ jobs:
           cache-dependency-path: pyproject.toml
       - name: Install Dependencies
         if: steps.cache.outputs.cache-hit != 'true'
-        run: pip install .[nats,kafka,rabbit,docs,testing]
+        run: pip install .[nats,kafka,rabbit,redis,docs,testing]
       - run: mkdir coverage
       - name: Test
         run: bash scripts/test.sh -m "(slow and nats) or nats"
@@ -267,13 +267,62 @@ jobs:
       - name: Test
         run: bash scripts/test.sh -m "not nats" tests/brokers/nats/test_test_client.py
 
+  test-redis-real:
+    if: github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    services:
+      nats:
+        image: redis:alpine
+        ports:
+          - 6379:6379
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+          cache: "pip"
+          cache-dependency-path: pyproject.toml
+      - name: Install Dependencies
+        if: steps.cache.outputs.cache-hit != 'true'
+        run: pip install .[nats,kafka,rabbit,redis,docs,testing]
+      - run: mkdir coverage
+      - name: Test
+        run: bash scripts/test.sh -m "(slow and redis) or redis"
+        env:
+          COVERAGE_FILE: coverage/.coverage.redis-py
+          CONTEXT: redis-py
+      - name: Store coverage files
+        uses: actions/upload-artifact@v3
+        with:
+          name: coverage
+          path: coverage
+
+  test-redis-smoke:
+    if: github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+          cache: "pip"
+          cache-dependency-path: pyproject.toml
+      - name: Install Dependencies
+        if: steps.cache.outputs.cache-hit != 'true'
+        run: pip install .[redis,test-core]
+      - name: Test
+        run: bash scripts/test.sh -m "not redis" tests/brokers/redis/test_test_client.py
+
   coverage-combine:
     if: github.event.pull_request.draft == false
     needs:
       - test
       - test-kafka-real
       - test-rabbit-real
       - test-nats-real
+      - test-redis-real
     runs-on: ubuntu-latest
 
     steps:
@@ -314,11 +363,13 @@ jobs:
       - test-macos-latest
       - test-windows-latest
       - test-kafka-real
-      - test-rabbit-real
-      - test-nats-real
       - test-kafka-smoke
+      - test-rabbit-real
       - test-rabbit-smoke
+      - test-nats-real
       - test-nats-smoke
+      - test-redis-real
+      - test-redis-smoke
 
     runs-on: ubuntu-latest
 

diff --git a/.secrets.baseline b/.secrets.baseline
@@ -128,7 +128,7 @@
         "filename": "docs/docs/en/release.md",
         "hashed_secret": "35675e68f4b5af7b995d9205ad0fc43842f16450",
         "is_verified": false,
-        "line_number": 134,
+        "line_number": 136,
         "is_secret": false
       }
     ],
@@ -158,10 +158,10 @@
         "filename": "faststream/rabbit/broker.py",
         "hashed_secret": "35675e68f4b5af7b995d9205ad0fc43842f16450",
         "is_verified": false,
-        "line_number": 57,
+        "line_number": 58,
         "is_secret": false
       }
     ]
   },
-  "generated_at": "2023-11-24T08:08:28Z"
+  "generated_at": "2023-11-30T09:12:40Z"
 }
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,15 +1,5 @@
 > **_NOTE:_**  This is an auto-generated file. Please edit docs/docs/en/getting-started/contributing/CONTRIBUTING.md instead.
 
----
-# 0.5 - API
-# 2 - Release
-# 3 - Contributing
-# 5 - Template Page
-# 10 - Default
-search:
-  boost: 3
----
-
 # Development
 
 After cloning the project, you'll need to set up the development environment. Here are the guidelines on how to do this.
@@ -46,17 +36,17 @@ After activating the virtual environment as described above, run:
 pip install -e ".[dev]"
 ```
 
-This will install all the dependencies and your local FastStream in your virtual environment.
+This will install all the dependencies and your local **FastStream** in your virtual environment.
 
-### Using Your local FastStream
+### Using Your local **FastStream**
 
-If you create a Python file that imports and uses FastStream, and run it with the Python from your local environment, it will use your local FastStream source code.
+If you create a Python file that imports and uses **FastStream**, and run it with the Python from your local environment, it will use your local **FastStream** source code.
 
-Whenever you update your local FastStream source code, it will automatically use the latest version when you run your Python file again. This is because it is installed with `-e`.
+Whenever you update your local **FastStream** source code, it will automatically use the latest version when you run your Python file again. This is because it is installed with `-e`.
 
 This way, you don't have to "install" your local version to be able to test every change.
 
-To use your local FastStream CLI, type:
+To use your local **FastStream CLI**, type:
 
 ```bash
 python -m faststream ...
@@ -66,7 +56,7 @@ python -m faststream ...
 
 ### Pytest
 
-To run tests with your current FastStream application and Python environment, use:
+To run tests with your current **FastStream** application and Python environment, use:
 
 ```bash
 pytest tests
@@ -82,6 +72,7 @@ In your project, you'll find some *pytest marks*:
 * **rabbit**
 * **kafka**
 * **nats**
+* **redis**
 * **all**
 
 By default, running *pytest* will execute "not slow" tests.
@@ -95,7 +86,7 @@ pytest -m 'all'
 If you don't have a local broker instance running, you can run tests without those dependencies:
 
 ```bash
-pytest -m 'not rabbit and not kafka and not nats'
+pytest -m 'not rabbit and not kafka and not nats and not redis'
 ```
 
 To run tests based on RabbitMQ, Kafka, or other dependencies, the following dependencies are needed to be started as docker containers:
@@ -140,6 +131,14 @@ services:
     # https://semgrep.dev/r?q=yaml.docker-compose.security.no-new-privileges.no-new-privileges
     security_opt:
       - no-new-privileges:true
+  # nosemgrep: yaml.docker-compose.security.writable-filesystem-service.writable-filesystem-service
+  redis:
+    image: redis:alpine
+    ports:
+      - 6379:6379
+    # https://semgrep.dev/r?q=yaml.docker-compose.security.no-new-privileges.no-new-privileges
+    security_opt:
+      - no-new-privileges:true
 ```
 
 You can start the dependencies easily using provided script by running:

diff --git a/README.md b/README.md
@@ -59,7 +59,7 @@ parsing, networking and documentation generation automatically.
 
 Making streaming microservices has never been easier. Designed with junior developers in mind, **FastStream** simplifies your work while keeping the door open for more advanced use cases. Here's a look at the core features that make **FastStream** a go-to framework for modern, data-centric microservices.
 
-- **Multiple Brokers**: **FastStream** provides a unified API to work across multiple message brokers (**Kafka**, **RabbitMQ**, **NATS**, support)
+- **Multiple Brokers**: **FastStream** provides a unified API to work across multiple message brokers (**Kafka**, **RabbitMQ**, **NATS**, **Redis** support)
 
 - [**Pydantic Validation**](#writing-app-code): Leverage [**Pydantic's**](https://docs.pydantic.dev/) validation capabilities to serialize and validates incoming messages
 
@@ -102,6 +102,8 @@ pip install faststream[kafka]
 pip install faststream[rabbit]
 # or
 pip install faststream[nats]
+# or
+pip install faststream[redis]
 ```
 
 By default **FastStream** uses **PydanticV2** written in **Rust**, but you can downgrade it manually, if your platform has no **Rust** support - **FastStream** will work correctly with **PydanticV1** as well.
@@ -129,10 +131,12 @@ from faststream import FastStream
 from faststream.kafka import KafkaBroker
 # from faststream.rabbit import RabbitBroker
 # from faststream.nats import NatsBroker
+# from faststream.redis import RedisBroker
 
 broker = KafkaBroker("localhost:9092")
 # broker = RabbitBroker("amqp://guest:guest@localhost:5672/")
 # broker = NatsBroker("nats://localhost:4222/")
+# broker = RedisBroker("redis://localhost:6379/")
 
 app = FastStream(broker)
 

diff --git a/docs/create_api_docs.py b/docs/create_api_docs.py
@@ -252,6 +252,7 @@ def _generate_api_docs_for_module(root_path: Path, module_name: str) -> str:
     api_summary = _get_api_summary(members_with_submodules)
 
     api_root = root_path / "docs" / "en" / "api"
+    api_root.mkdir(parents=True, exist_ok=True)
 
     (api_root / ".meta.yml").write_text(API_META)
 
@@ -274,9 +275,9 @@ def create_api_docs(
     docs_dir = root_path / "docs"
 
     # read summary template from file
-    summary_template = (docs_dir / "summary_template.txt").read_text()
+    navigation_template = (docs_dir / "navigation_template.txt").read_text()
 
-    summary = summary_template.format(api=api)
+    summary = navigation_template.format(api=api)
 
     summary = "\n".join(filter(
         bool,