[Self-Host] excludeTags should work with onlyMainContent?

[piotrstarzynski]

Hello,

Issue: Support excludeTags with onlyMainContent=true

Current Behavior

When onlyMainContent is set to true, the excludeTags parameter appears to be ignored. The content extraction works, but I cannot apply additional tag exclusions.

Expected Behavior

excludeTags should work in conjunction with onlyMainContent=true, allowing for:

• Base content filtering through onlyMainContent
• Additional fine-tuned control through custom excludeTags

Problem Details

Currently:

1. With onlyMainContent=true: excludeTags are ignored
2. With onlyMainContent=false: using excludeTags from the default list (https://github.com/mendableai/firecrawl/blob/79e65f31ef1d7a4172870471d81501ee2e8aef22/apps/api/src/scraper/WebScraper/utils/excludeTags.ts) plus custom tags results in longer output

This suggests that onlyMainContent uses a more aggressive content filtering algorithm than just tag exclusion.

Proposed Solution

Add support for excludeTags to work even when onlyMainContent=true. This would provide more flexible content filtering options by combining both features.

Use Case

This would be useful when users want to:

1. Get the main content of a page (onlyMainContent=true)
2. Additionally exclude specific tags that might still be present in the main content

Would it be possible to implement this feature enhancement?

[nickscamara]

@piotrstarzynski I believe this was fixed! Can you double check and confirm? Thank you!

[piotrstarzynski]

@nickscamara, I checked it once again and I believe it still doesn't work. In the example below, I still get a div with id=comments. Could someone please also confirm it?

        payload = {
            "url": url,
            "formats": ["markdown"],  # Request markdown format
            "pageOptions": {
                "onlyMainContent": True,
                "replaceAllPathsWithAbsolutePaths": True
            },
            "excludeTags": ["#comments"],
            #"includeTags": ["p"],
            "waitFor": 5000,
            "timeout": 50000
        }