Validate suite method signatures and continue execution for valid tests

[vyas-git]

Summary

This PR improves method signature validation in Testify's suite execution by ensuring that invalid methods are logged, and valid methods continue to execute.

Changes

• Added validation of method signatures (checking for arguments and return values) at an earlier stage, after method filtering.
• Replaced t.Fatalf with t.Errorf to report invalid method signatures while allowing the test suite to continue executing valid methods.
• Introduced additional test cases to cover both valid and invalid method signatures.

Motivation

• To ensure that invalid method signatures in the suite are reported without stopping execution, allowing valid methods to run and improving overall test suite behavior.

Related issues

• Closes suite: missing signature check before calling Test* method #1508

[brackendawson]

This error shows up when you run a valid test. Previously there is no panic in this scenario.

eg:

package kata_test

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/suite"
)

type ExampleTestSuite struct {
	suite.Suite
	VariableThatShouldStartAtFive int
}

func (suite *ExampleTestSuite) SetupTest() {
	suite.VariableThatShouldStartAtFive = 5
}

func (suite *ExampleTestSuite) TestExample() {
	assert.Equal(suite.T(), 6, suite.VariableThatShouldStartAtFive)
}

func (suite *ExampleTestSuite) TestSomething(a any) {
	suite.Equal(1, a)
}

func TestExampleTestSuite(t *testing.T) {
	suite.Run(t, new(ExampleTestSuite))
}

% go test -run=TestExampleTestSuite/TestExample
--- FAIL: TestExampleTestSuite (0.00s)
    suite.go:153: testify: suite method 'TestSomething' has 1 input arguments and 0 return values. It should have none.
FAIL
exit status 1
FAIL    github.com/brackendawson/kata   0.280s

[vyas-git]

@brackendawson Thank you for pointing that out! I understand the concern.

Since t is tied to the suite instance, this caused the entire suite to be marked as failed whenever an invalid method was detected—even when running a specific test like TestExampleTestSuite/TestExample.

The suite considers all its methods, and methodFilter cannot filter down to a specific test.

I considered using t.Skip within a testing.InternalTest case, but creating extra tests for invalid methods just to call t.Skip felt unnecessary. Skipping invalid methods early with a warning message turned out to be a cleaner solution.

To address this, I updated the approach to log a warning message directly and skip invalid methods without using t.Errorf. This change ensures:

1. Invalid methods are skipped, but valid methods continue to execute without marking the entire suite as failed.
2. Only the requested test methods are executed, while the suite logs a warning message for any invalid method signatures encountered.

Thank you for your feedback, please let me know if this approach aligns with the intended behavior or if there are further adjustments you’d like to see!