Reply to thread

Message: [QUOTE="Bombastus, post: 90989, member: 2178"] Pluginval can run validation from strictness level 1 through 10, with level 5 generally treated as the minimum for host compatibility. Higher settings do not simply repeat the same scan more times. They add tougher state, threading, automation, and processing conditions that a plugin can survive in ordinary use without ever encountering during a quick launch test. CarbonDSP says every shipped plugin is checked at level 10 before release, but that badge needs the right interpretation. A successful run means the tested build survived pluginval's enabled tests under that configuration. It does not mean every DAW, driver, operating-system build, project layout, or hardware combination has been exercised. [HEADING=2]Higher strictness makes ordinary operations nastier[/HEADING] At low levels, pluginval covers fundamentals such as opening the plugin, reading its information, changing programs, creating the editor, processing audio, restoring state, and checking parameters. Audio processing is also tried across configured sample rates and block sizes, with invalid numerical output such as NaNs and infinities treated as failures. Move above level 5 and several existing tests harsher. Editor automation grows from 100 random parameter-change passes to 1,000, while automation processing treats subnormal output as an error rather than merely logging it. VST3 validation can also use the validator's extended mode when that integration is enabled. Level 6 adds behavior that many simple smoke tests never touch. Pluginval changes sample rate without first calling the normal resource-release step, then checks whether processing still behaves. Its state-restoration test randomizes parameters, restores the saved state, and verifies that values return where they belong. At level 8, that same state test becomes stricter again. Pluginval reads the restored state back and requires an exact binary match with the state that the host supplied. That can expose a plugin that looks correct on screen after recall while quietly serializing its state differently on the next save. [HEADING=2]Threading tests target ugly host behavior[/HEADING] Level 7 introduces some of the more revealing cases because hosts do not always call plugins in the neat sequence developers expect. One test opens the editor on the message thread while state is read, parameters are randomized, and saved state is restored from a background thread. Another test attacks parameter handling from two directions at once. The message thread repeatedly sends host-notified parameter changes while another thread changes parameters, processes audio, and runs hundreds of iterations. That [B]concurrent automation and state pressure[/B] is designed to expose race conditions that may stay invisible during normal knob turning. This matters because a plugin can sound perfect and still fail as software. A race around parameter state, an unsafe editor interaction, or a broken restore path can surface only after a host automates controls, reloads a session, or performs work off the main thread. Those are reliability defects rather than audio-quality defects. The [B][URL='https://goldmidi.com/community/threads/carbondsp-introduced-four-audio-plugins.76243/']CarbonDSP release validation process[/URL][/B] therefore carries more weight than a claim that the plugins merely open successfully. Level 10 places the real build through the highest strictness available in pluginval, including the extended tests reached on the way up through the lower levels. [HEADING=2]A level 10 pass still has boundaries[/HEADING] Pluginval is a test suite, not a universal compatibility certificate. Its own project guidance describes level 5 as the usual minimum for host compatibility, while higher levels contain longer and more aggressive tests. Passing level 10 proves that the plugin passed those tests, not that every possible host interaction has been mathematically exhausted. Platform details matter too. Pluginval's current project notes describe real-time safety checking as available on macOS through its rtcheck integration, so the phrase level 10 should not be read as an identical low-level test environment on every operating system. Build options can also affect whether format-specific validation components are present. A failing result needs interpretation for the same reason. The current pluginval issue tracker still contains an open report where the parameter thread-safety test can time out on heavier processing, illustrating that a validator failure may require debugging before you know whether the plugin, the test conditions, or the validator itself caused it. CarbonDSP does not rely on pluginval alone. Its release process also includes an offline render-smoke pass, a headless UI snapshot, and testing of plugin and standalone builds across different machines after the automated gate clears. Those checks cover failure classes that a single validator run cannot, especially visual regressions, direct DSP rendering problems, and machine-specific behavior. That extra machine pass extends beyond one automated validator configuration. [/QUOTE]

Name