[Bug]: Go SDK pipeline options define global flags with generic names, causing flag redefinition panics #40730

What happened?

The Beam Go SDK defines pipeline options as global flags on Go's default flag.CommandLine at package initialization time (options/jobopts, options/gcpopts, runners/dataflow, core/runtime/harness/init, etc.). Many of these flags use generic, unprefixed names (--endpoint, --job_name, --async, --experiments, --parallelism, --project, --region, --id, --options, etc.).
Because Go's flag package panics when duplicate flags are defined (panic: flag redefined: <name>), any application or test binary that defines flags with the same names crashes at startup if its dependency graph pulls in these Beam packages.

Recent Regressions (#40379 and #40686)

Previously, options/jobopts and options/gcpopts were only linked when using specific runners or x/beamx. Two recent changes caused core and utility packages to depend on them, triggering flag redefined panics in downstream test suites:

1. [Claude] Plumb through pipeline-level resource hints to cross-language transforms #40379: core/runtime/xlangx/expand.go added a dependency on options/jobopts (for jobopts.GetPipelineResourceHints()). Because the root beam package depends on xlangx, linking github.com/apache/beam/sdks/v2/go/pkg/beam now registers 13 global flags (--endpoint, --async, --job_name, etc.).
2. [Go] Support non-default universes for GCS and Dataflow clients #40686: util/gcsx/gcs.go added a dependency on options/gcpopts (for gcpopts.GetUniverseDomain() and gcpopts.GetGcsEndpoint()). Linking util/gcsx (or io/filesystem/gcs) now registers --project, --region, and --gcs_endpoint globally.

Reproduction

package mypkg_test
import (
	"flag"
	"testing"
	_ "github.com/apache/beam/sdks/v2/go/pkg/beam"
	_ "github.com/apache/beam/sdks/v2/go/pkg/beam/util/gcsx"
)
var (
	endpoint = flag.String("endpoint", "", "custom endpoint") // panics after #40379: "flag redefined: endpoint"
	project  = flag.String("project", "", "custom project")   // panics after #40686: "flag redefined: project"
)
func TestSomething(t *testing.T) {}

Possible Solutions

1. Short-term: Avoid depending on options/jobopts and options/gcpopts from core/utility packages (core/runtime/xlangx, util/gcsx), or decouple option retrieval from global flag registration.
2. Long-term: Move away from unconditional init()-time registration on flag.CommandLine—for example, by requiring explicit flag registration (e.g., in beamx.Run) or supporting structured pipeline options passed directly to runners.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner