feat: write-path content-hash dedup for MemoryManager.add

[Zhe-SH-CN]

Description

MemOS has no server-side dedup on the write path. Any client that re-sends identical memory content (retry loops, replayed events, buggy integrations) creates duplicate nodes that pollute search results, graph structure, and reorganization.

This PR adds content-hash deduplication at MemoryManager.add:

1. Compute sha1(memory_text) for each incoming TextualMemoryItem
2. Dedup within the batch using a set
3. Query graph_store.get_by_metadata for existing nodes with the same content_hash
4. Skip items whose hash exists within a configurable time window (default 7 days)
5. Fail-open: if the dedup check errors, the write proceeds (availability over strictness)
6. Persist content_hash as a metadata property on the graph node
7. Toggles: MOS_DEDUP_ENABLED=0 to disable, MOS_DEDUP_WINDOW_DAYS to adjust the window

Related Issue: Fixes #2141

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested

• Unit tests covering batch-level dedup, graph-store-level dedup, time window logic, fail-open behavior, empty input, and mixed duplicates
• python3 -c "import ast; ast.parse(...)" syntax verification

Checklist

• I have performed a self-review of my own code
• I have commented my code in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have linked the issue to this PR

[Memtensor-AI]

🤖 Open Code Review

Target: PR #2153
Task: 258da3fb72ae3edf
Base: dev-v2.0.24
Head: feat/write-path-content-hash-dedup

🔍 OpenCodeReview found 3 issue(s) in this PR.

⚠️ 1 warning(s) occurred during review.

---

1. src/memos/memories/textual/tree_text_memory/organize/manager.py (L30)

Unguarded int() conversion of an environment variable will raise ValueError at module import time if MOS_DEDUP_WINDOW_DAYS is set to a non-integer value (e.g., "7.5", "one week", or an empty string ""), crashing the whole application startup. Wrap it in a try/except or use a helper that falls back to the default.

💡 Suggested Change

Before:

_DEDUP_WINDOW_DAYS = int(os.getenv("MOS_DEDUP_WINDOW_DAYS", "7"))

After:

try:
    _DEDUP_WINDOW_DAYS = int(os.getenv("MOS_DEDUP_WINDOW_DAYS", "7"))
except ValueError:
    logger.warning(
        "Invalid MOS_DEDUP_WINDOW_DAYS value %r; falling back to 7 days",
        os.getenv("MOS_DEDUP_WINDOW_DAYS"),
    )
    _DEDUP_WINDOW_DAYS = 7

---

2. src/memos/memories/textual/tree_text_memory/organize/manager.py (L152)

Timezone-aware vs naive datetime mismatch. datetime.now() returns a naive datetime, but datetime.fromisoformat(updated) returns a timezone-aware datetime when the stored string includes a UTC offset (e.g., "2026-07-20T10:00:00+00:00"). Comparing the two raises TypeError, which is swallowed by the outer except Exception block and treated as fail-open — silently disabling dedup for every node that has a timezone-aware timestamp. Use datetime.now(timezone.utc) and ensure the parsed timestamp is also timezone-aware.

💡 Suggested Change

Before:

                            if ts and ts >= cutoff:

After:

from datetime import datetime, timedelta, timezone

# ...
cutoff = datetime.now(timezone.utc) - timedelta(days=_DEDUP_WINDOW_DAYS)
# ...
                            try:
                                ts = datetime.fromisoformat(updated)
                                # Make naive timestamps UTC-aware for a safe comparison
                                if ts.tzinfo is None:
                                    ts = ts.replace(tzinfo=timezone.utc)
                            except (ValueError, TypeError):
                                ts = None
                            if ts and ts >= cutoff:

---

3. src/memos/memories/textual/tree_text_memory/organize/manager.py (L169-L176)

The fallback branch m.metadata.info = {"content_hash": h} silently replaces the entire info dict with a new single-key dict, destroying any pre-existing metadata that was stored there. TextualMemoryMetadata.info is a real dict | None field used for arbitrary key-value metadata. If m.metadata.info is not a dict (e.g., None), the correct fix is to initialize it as {"content_hash": h}, but if it already contains data, only the content_hash key should be added or updated.

Additionally, since TextualMemoryMetadata uses model_config = ConfigDict(extra="allow"), the primary m.metadata.content_hash = h assignment on Pydantic models will always succeed — making the entire except block unreachable dead code.

💡 Suggested Change

Before:

            try:
                m.metadata.content_hash = h
            except (AttributeError, TypeError):
                # Metadata model may not support direct attribute set.
                if hasattr(m.metadata, "info") and isinstance(m.metadata.info, dict):
                    m.metadata.info["content_hash"] = h
                else:
                    m.metadata.info = {"content_hash": h}

After:

            # TextualMemoryMetadata uses extra="allow", so direct assignment always works.
            m.metadata.content_hash = h

Generated by cloud-assistant via Open Code Review.

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (8/8 executed). memos_python_core/changed-repo-python: 8/8. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-258da3fb72ae3edf-20260723200033: 112/120 passed, 8 failed — these do NOT affect the PR verdict; review the branch manually.

Branch: feat/write-path-content-hash-dedup

[WeiminLee]

ok