Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid lambda in _config module level code #860

Merged

Conversation

jmartin-tech
Copy link
Collaborator

When using --parallel_attempts some probes will fail to pickle the Probe._execute_attempt method due to garak._config containing a lambda that is evaluated at import time.

By providing a concrete method call to replace the lambda the pickle is resolved.

The issue can be triggered with nvcf:

% python -m garak -m nvcf -n nvidia/nemotron-4-340b-instruct -p lmrc --parallel_attempts 5
garak LLM vulnerability scanner v0.9.0.15.post1 ( https://github.com/leondz/garak ) at 2024-08-28T16:34:08.687066
📜 logging to /Users/jemartin/.local/share/garak/garak.log
🦜 loading generator: NVCF: nvidia/nemotron-4-340b-instruct
📜 reporting to /Users/jemartin/.local/share/garak/garak_runs/garak.48b9d57d-442a-4758-8d6a-68ebaacac50d.report.jsonl
🕵️  queue of probes: lmrc.Anthropomorphisation, lmrc.Bullying, lmrc.Deadnaming, lmrc.Profanity, lmrc.QuackMedicine, lmrc.SexualContent, lmrc.Sexualisation, lmrc.SlurUsage
probes.lmrc.Anthropomorphisation:   0%|                                                                | 0/6 [00:00<?, ?it/s]Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jemartin/Projects/nvidia/garak/garak/__main__.py", line 14, in <module>
    main()
  File "/Users/jemartin/Projects/nvidia/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/Users/jemartin/Projects/nvidia/garak/garak/cli.py", line 502, in main
    command.probewise_run(
  File "/Users/jemartin/Projects/nvidia/garak/garak/command.py", line 222, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/Users/jemartin/Projects/nvidia/garak/garak/harnesses/probewise.py", line 107, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/Users/jemartin/Projects/nvidia/garak/garak/harnesses/base.py", line 108, in run
    attempt_results = probe.probe(model)
                      ^^^^^^^^^^^^^^^^^^
  File "/Users/jemartin/Projects/nvidia/garak/garak/probes/base.py", line 211, in probe
    attempts_completed = self._execute_all(attempts_todo)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jemartin/Projects/nvidia/garak/garak/probes/base.py", line 174, in _execute_all
    for result in attempt_pool.imap_unordered(
  File "/Users/jemartin/.pyenv/versions/3.12.4/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
  File "/Users/jemartin/.pyenv/versions/3.12.4/lib/python3.12/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/Users/jemartin/.pyenv/versions/3.12.4/lib/python3.12/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jemartin/.pyenv/versions/3.12.4/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x1014ef060>: attribute lookup <lambda> on garak._config failed

@jmartin-tech jmartin-tech added bug Something isn't working quality-speed This affects the speed of program use labels Aug 28, 2024
Copy link
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's so rare that pickle makes things /more/ beautiful

@jmartin-tech jmartin-tech merged commit 47299fb into NVIDIA:main Aug 29, 2024
8 checks passed
@jmartin-tech jmartin-tech deleted the fix/remove-config-module-lambda branch August 29, 2024 13:01
@github-actions github-actions bot locked and limited conversation to collaborators Aug 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working quality-speed This affects the speed of program use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants